Variable resolution eye mounted displays

ABSTRACT

A display device (e.g., in a contact lens) is mounted on the eye. The eye mounted display contains multiple sub-displays, each of which projects light to different retinal positions within a portion of the retina corresponding to the sub-display. Additionally, a “locally uniform resolution” mapping may be used to model the variable resolution of the eye. Accordingly, various aspects of the display device may be based on the locally uniform resolution mapping. For example, the light emitted from the sub-displays may be based on the locally uniform resolution mapping.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/590,056, filed Jan. 6, 2015, now U.S. Pat. No. 9,993,335; whichclaims the benefit of U.S. Provisional Application No. 61/924,924, filedJan. 8, 2014, which are both incorporated by reference in theirentirety.

This application relates to U.S. patent application Ser. No. 12/359,211,“Eye Mounted Displays,” by Michael Deering and Alan Huang, filed on Jan.23, 2009, and U.S. patent application Ser. No. 12/359,951, “SystemsUsing Eye Mounted Displays,” by Michael Deering, filed on Jan. 26, 2009.The subject matter of all of the foregoing is incorporated herein byreference in their entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates generally to visual display technology. Moreparticularly, it relates to display technology for eye mounted display,and specifically to contact lens displays.

2. Description of the Related Art

More and more our technological society relies on visual displaytechnology for work, home, and on-the-go use: document and productivityapplications, text messages, email, internet access, voice calls, games,HDTV, video on demand, still and video cameras, etc. These services aredelivered over different devices that mainly vary in size, for example,smartphones, dedicated digital cameras, dedicated portable game devices,tablets, laptops, desktops, TV sets, game consoles, and servers(including the cloud). What these all (but the last) have in common is adedicated display, mainly constrained by the physical size of theparticular device. Restrictions in size limit the possible field ofview, which limits not just the resolution but scale of information andcontext available. Larger physical display sizes alleviate such limits,but rule out portability, and increases power requirements as well ascost. While there is some convergence in devices, e.g. smartphones arecommonly used instead of dedicated small digital cameras, physicaldisplay size currently limits how far such convergence can go. Forexample, the electronics of a modern smartphone, tablet, and low powerlaptop are effectively identical, except for the physical display deviceand the incremental additional physical battery size required to powerit. Thus there is a need for improvements in display technologies withrespect to spatial resolution, quality, field of view, portability (bothsize and power consumption), cost, etc.

However, the current crop of display technologies makes a number oftradeoffs between these goals in order to satisfy a particular marketsegment. For example, direct view color CRTs do not allow directaddressing of individual pixels. Instead, a Gaussian spread out overseveral phosphor dots (pixels) both vertically and horizontally(depending on spot size) results. Direct view LCD panels have generallyreplaced CRTs in most computer display and TV display markets, but atthe trade-offs of temporal lag in sequences of images, lower colorquality, lower contrast, and limitations on viewing angles. Displaydevices with resolutions higher than the 1920×1024 HDTV standards arenow available, but at substantially higher cost. The same is true fordisplays with higher dynamic range or high frame rates. Projectiondisplay devices can now produce large, bright images, but at substantialcosts in lamps and power consumption. Displays for cell phones, tablets,handheld games, small still and video cameras, etc., must currentlyseriously compromise resolution and field of view. Within thespecialized market where head mounted displays are used, there are stillserious limitations in resolution, field of view, undo warpingdistortion of images, weight, portability, cosmetic acceptability, andcost.

The existing technologies for providing direct view visual displaysinclude CRTs, LCDs, OLEDs, OLED on silicon, LEDs, plasma, SEDs, liquidpaper, etc. The existing technologies for providing front or rearprojection visual displays include CRTs, LCDs, DLP™, LCOS, linear MEMsdevices, scanning laser, etc. All these approaches have much highercosts when higher light output is desired, as is necessary when largerdisplay surfaces are desired, when wider useable viewing angles aredesired, for stereo display support, etc.

Another general problem with current direct view display technology isthat they are all inherently limited in the perceivable resolution andfield of view that they can provide when embedded in small portableelectronics products. Only in laptop computers (which are quite bulkycompared to cell phones, tablets, hand held game systems, or small stilland/or video cameras) can one obtain higher resolution and field of viewin exchange for size, weight, cost, battery weight and life time betweencharges. Larger, higher resolution direct view displays are bulky enoughthat they must remain in the same physical location day to day (e.g.,large plasma, LCD, or OLED display devices).

One problem with current rear projection display technologies is thatthey tend to come in very heavy bulky cases to hold folding mirrors. Andto compromise on power requirement and lamp cost most use display screentechnology that preferentially passes most of the light over a narrowrange of viewing angles.

One problem with current front projection display technology is thatthey take time to set up, usually need a large external screen, andwhile some are small enough to be considered portable, the weightsavings comes at the price of color quality, resolution, and maximumbrightness. Many also have substantial noise generated by their coolingfans.

Current head mounted display technology have limitations with respect toresolution, field of view, image linearity, weight, portability, andcost. They either make use of display devices designed for other largermarkets (e.g., LCD devices for video projection), and put up with theirlimitations; or custom display technologies must be developed for whatis still a very small market. While there have been many innovativeoptical designs for head mounted displays, controlling the light fromthe native display to the device's exit pupil can result in bulky, heavyoptical designs, and rarely can see-through capabilities (for augmentedreality applications, etc.) be achieved. While head mounted displaysrequire lower display brightness than direct view or projectiontechnologies, they still require relatively high display brightnessbecause head mounted displays must support a large exit pupil to coverrotations of the eye, and larger stand-off requirements, for example toallow the wearing of prescription glasses under the head mounteddisplay.

Thus, there is a need for new display technologies to overcome theresolution, field of view, power requirements, bulk and weight, lack ofstereo support, frame rate limitations, image linearity, and/or costdrawbacks of present display technologies

BRIEF DESCRIPTION OF THE DRAWINGS

The invention has other advantages and features which will be morereadily apparent from the following detailed description of theinvention and the appended claims, when taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is an electronic device desiring visual output, an eye mounteddisplay controller, an eye mounted display, and a human user.

FIG. 2 is an eye mounted display system.

FIG. 3 is a large configuration of an eye mounted display system.

FIG. 4 is a block diagram of a scaler.

FIG. 5 (prior art) is a two dimensional cross section of the threedimensional human eye.

FIG. 6 (prior art) is a zoom into the corneal portion of the human eye.

FIG. 7 (prior art) is a zoom into the foveal region of the retinalportion of the human eye.

FIG. 8 (prior art) is a two dimensional vertical cross section of thethree dimensional human eye.

FIG. 9 shows a physical world rock cliff being observed by a human.

FIG. 10 shows two conventional flat panel displays being observed by ahuman.

FIG. 11 shows a conventional head mounted display being observed by ahuman.

FIG. 12 shows a zoomed view of FIG. 11 to show the details of the changein radius spherical waves.

FIG. 13 is a variation of FIG. 5, in which the two optical elements ofthe human eye have been highlighted.

FIG. 14 is a variation of FIG. 9, in which only the point source and thespherical wavefronts of light it generates and the human observer's eyeare shown.

FIG. 15 is a variation of FIG. 14, in which the wavefronts of light asmodified by the optical effects of the cornea have been added.

FIG. 16 is a variation of FIG. 15, in which the wavefronts of light asmodified by the optical effects of the lens have been added.

FIG. 17 is a variation of FIG. 16, in which the point on the retinalsurface where the wavefronts of light converge is shown.

FIG. 18 is a variation of FIG. 13, in which a contact lens is beingworn, and in which the resultant two (major) optical elements of thecontact lens eye system have been highlighted.

FIG. 19 shows a femto projector embedded in a contact lens andgenerating post-corneal wavefronts of light.

FIG. 20 is a zoom into FIG. 19 and shows detail of a femto projectorgenerating post-corneal wavefronts of light.

FIG. 21 shows a hexagon on its side.

FIG. 22 shows a hexagon on its end.

FIG. 23 shows the long diagonals of a hexagon on its side.

FIG. 24 shows the long diagonals of a hexagon on its end.

FIG. 25 shows the short diagonals of a hexagon on its side.

FIG. 26 shows the short diagonals of a hexagon on its end.

FIG. 27 shows a tiling of hexagons on their side.

FIG. 28 shows a tiling of hexagons on their end.

FIG. 29 shows the long pitch of a hexagonal tiling.

FIG. 30 shows the short pitch of a hexagonal tiling.

FIG. 31 shows the even rows of a tiling of hexagons on their end.

FIG. 32 shows the odd rows of a tiling of hexagons on their end.

FIG. 33 shows the first even row of a tiling of hexagons on their end.

FIG. 34 shows the first odd row of a tiling of hexagons on their end.

FIG. 35 shows the first semi-column of a tiling of hexagons on theirend.

FIG. 36 shows the second semi-column of a tiling of hexagons on theirend.

FIG. 37 shows the semi-column numbers of the first row of a tiling ofhexagons on their end.

FIG. 38 shows the semi-column numbers of the second row of a tiling ofhexagons on their end.

FIG. 39 shows the integral addresses of a tiling of hexagons on theirend.

FIG. 40 shows the height of a unit-width hexagon in a tiling of hexagonson their end.

FIG. 41 shows the row to row pitch of a tiling of unit width hexagons ontheir end.

FIG. 42 shows a 1-group of hexagons their end

FIG. 43 shows a 2-group of hexagons their end

FIG. 44 shows a 3-group of hexagons their end

FIG. 45 shows a 4-group of hexagons their end

FIG. 46 shows a hexagonal tiling of three 2-groups of hexagons on theirend.

FIG. 47 shows a square on its side.

FIG. 48 shows a square on its end.

FIG. 49 shows the long diagonals of a square on its side.

FIG. 50 shows the long diagonals of a square on its end.

FIG. 51 shows the short diagonals of a square on its side.

FIG. 52 shows the short diagonals of a square on its end.

FIG. 53 shows the long pitch of a square tiling.

FIG. 54 shows the short pitch of a square tiling.

FIG. 55 shows the locally uniform resolution mapping.

FIG. 56 shows 28 hexagonally shaped projectors in 7 circles of 4.

FIG. 57 shows 35 hexagonally shaped projectors in 7 circles of 5.

FIG. 58 shows 42 hexagonally shaped projectors in 7 circles of 6.

FIG. 59 shows 49 hexagonally shaped projectors in 7 circles of 7.

FIG. 60 shows 56 hexagonally shaped projectors in 7 circles of 8.

FIG. 61 shows the foveal region of variable resolution projectors, thena combination of variable plus 7 fixed resolution projectors.

FIG. 62 shows the foveal region of a combination of variable and 13fixed resolution projectors.

FIG. 63 shows a combination 30 variable resolution plus 13 fixedresolution projectors

FIG. 64 shows a tiling of 49 and 55 hexagonally shaped projectors.

FIG. 65 shows a tiling of 61 and 67 hexagonally shaped projectors.

FIG. 66 shows a 3-group (7 hexagon wide) fixed resolution 37 pixelprojector.

FIG. 67 shows a 7-group (15 hexagon wide) fixed resolution 169 pixelprojector.

FIG. 68 shows a 15-group (31 hexagon wide) fixed resolution 721 pixelprojector.

FIG. 69 shows a 31-group (63 hexagon wide) fixed resolution 2,977 pixelprojector.

FIG. 70 shows a 63-group (127 hexagon wide) fixed resolution 12,097pixel projector.

FIG. 71 shows a 5-group (11 hexagon wide) variable resolution 91 pixelprojector.

FIG. 72 shows a 10-group (21 hexagon wide) variable resolution 331 pixelprojector.

FIG. 73 shows a 20-group (41 hexagon wide) variable resolution 1,261pixel projector.

FIG. 74 shows a 40-group (81 hexagon wide) variable resolution 4,921pixel projector.

FIG. 75 shows a single femto projector.

FIG. 76 shows a single femto projector constructed from four pieces.

FIG. 77 shows a single femto projector constructed from six pieces.

FIG. 78 shows a cross-section through a contact lens display showingthree foveal and six peripheral femto projectors as cylinders.

FIG. 79 shows a cross-section through a contact lens display showingthree foveal and six peripheral femto projector optics.

FIG. 80 shows a top view of the display mesh.

FIG. 81 shows a top view of the display mesh with the outer ring IC die.

FIG. 82 shows a zoomed top view of the display mesh with the cornerreflector detail.

FIG. 83 shows an aligned top and side view of the display mesh.

FIG. 84 shows the display mesh display controller die data to individualfemto projector display die wiring paths.

FIG. 85 shows the individual femto projector display die common databack to display controller die wiring path.

FIG. 86 shows details of the display mesh display controller die data toindividual femto projector display die wiring paths.

FIG. 87 shows the femto projector blocking portions of contact lensoptical zone.

FIG. 88 shows the 43 femto projectors of FIG. 87 individually numbered.

FIG. 89 shows the ratio of cone short pitch models of Dacey and Taylor,from 0.5° to 6°.

FIG. 90 shows the ratio of the short pitch of the locally uniformresolution mapping to Dacey's model and the ratio of short pitch of theLogPolar mapping to Dacey's model.

FIG. 91 shows resolution curves from human testing, of the locallyuniform resolution mapping, of Dacey's model, and the resolution ofcones from Tyler's model.

FIG. 92 shows the relationships between several geometric points,distances, and angles, between the ViewPlane and the ViewSphere inspace.

All figures include an element with a number the same as the figurenumber except multiplied by 100: this is always the title element forthe figure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Outline

I. Overview

II. Anatomical and Visual Definitions

-   -   II.A. Terminology    -   II.B. Retinal Definitions    -   II.C. Axis: Classical Optical vs. the Human Eye    -   II.D. Important Retinal Neural Cell Types        III. Mathematical Definitions    -   III.A. Mathematical Conventions    -   III.B. Hexagonimania    -   III.C. Mathematical Definitions of the ViewSphere and Related        Concepts    -   III.D. Scale Changes between the ScreenSurface and ViewSpaceVS    -   III.E. Re-Definition of Resolution        IV. The Variable Resolution Nature of the Human Eye        V. Optical Function Required for Eye Mounted Displays        VI. Optical Function of a Contact Lens Display        VII. Variable Resolution Mappings for Contact Lens Displays        VIII. Means to Display Variable Resolution Pixels    -   VIII.A. Pixel Tilings Overview    -   VIII.B. Projector Tilings    -   VIII.C. Pixel Tilings Details    -   VIII.D. Sub-Sampling of Color for an Eye Mounted Display    -   VIII.E. Computer Graphics Real-Time Variable Resolution        Rendering    -   VIII.F. Generation of Discrete Pixel Component Values from the        Super-Sample Buffer    -   VIII.G. Computer Graphics Real-Time Rendering of Arbitrary        Pre-Inverse Distortion of Pixels    -   VIII.H. Using Computer Graphics Real-Time Rendering of Arbitrary        Pre-Inverse-Distortion of Pixels to Compensate for a Varity of        Optical and Manufacturing Based Distortions    -   VIII.I. Issues of Depth of Focus and Presbyopia.        IX. Physical Construction of the Display Mesh    -   IX.A. Material Considerations    -   IX.B. One Possible Solution: the Display Mesh    -   IX.C. Wiring up the Display Mesh    -   IX.D. Fabricating and Assembling the Display Mesh        X. Areas of Note

I. Overview

This patent relates to the optical elements of an eye mounted display,including how to construct a variable resolution display, how a computergraphics and/or video system can properly generate variable resolutionpixels for such a display, and how the optical and structural elementsmay be fabricated and assembled.

An eye mounted display is a display device that is mounted to someexterior or interior portion of the human eye, and rotates along withrotations of the eye. The display can be hard fixed to the eye, or slidearound some relative to the eye, much as current contact lenses do. Thedisplay can be mounted anywhere in the optical path of the human eye. Itcould be mounted offset from the cornea, allowing an air interfacebetween the display and the cornea. It could be mounted on top of thetear layer of the cornea (much as current contact lenses are). It couldbe mounted directly on top of the cornea (but then would have to providethe biological materials to maintain the cornea cells). It could bemounted inside of, in place of, or to the posterior of the cornea. Allof these mounting options discussed so far for eye mounted displayswould be also more narrowly classified as cornea mounted displays(CMD's).

The eye mounted display could be mounted within the aqueous humor,between the cornea and the crystalline lens, just as present so called“inter-ocular” lenses are. Such an eye mounted display would be morenarrowly defined as an inter-ocular mounted display.

Just as an eye mounted display could be mounted in front, inside,posterior to, or in place of the cornea, instead these options could beapplied to the crystalline lens. These would be lens mounted displays.

Finally, an eye mounted display could be mounted on the surface of theretina itself. In this one case many less optical components are needed;the display pixels are placed right above the cones to be displayed to.Although such a “retina mounted display” would have some issues with howto best surgically implant it; such a display might also act as apressure bandage and prevent (or slow down) macular degeneration.

For an eye mounted display to be effective, the overall display systemhas to know to high accuracy the orientation of the eye relative to thehead at all times. Several types of devices can provide such tracking;for the special case of cornea mounted displays fixed in positionrelative to the cornea, the problem devolves to the much simpler problemof tracking the orientation (and movement direction and velocity) of thecornea display; special fiducial marks on the surface of the corneamounted display can make this a fairly simple problem to solve. Othertypes of the eye mounted displays may require different solutions to theproblem of tracking the orientation of the eye to sufficient accuracy.

Eye mounted displays have two properties that generally external displaytechnologies cannot (easily) take advantage of. First, because thedisplay is mounted right on the cornea, only the amount of light thatactually will enter the eye (through the pupil) has to be produced.These savings are substantial; for an eye mounted display to produce theequitant retinal illumination as a 2,000 lumen video projector viewedfrom 8 feet away, the eye mounted display will only need to produce onone one-thousandth or less of a lumen: a factor of a million lessphotons.

A second inherent advantage of an eye mounted display is that the numberof “pixels” that have to be generated can be matched to the much lowernumber of cones (and effective cone groupings in the periphery)(approximately 400,000) as opposed to having to produce the highestfoveal resolution everywhere on the external display surface.(Technically, what has to be matched is the number of retinal midgetganglion neuron cells, of the “ON” type.) As an example, an eye mounteddisplay with only 400,000 physical pixels can produce imagery that anexternal display may need 100 million or more pixels to equal (a factorof 200 less pixels).

In modern times, the human eye is easiest described as a kind of videocamera—both video cameras and eyes capture photons over time impingingupon them from different directions in the physical world. But invirtually all man-made cameras (as well as displays), all pixels are thesame size: they have effectively the same resolution; while the “pixels”of the human eye (called retinal receptor fields) vary in area by afactor of more than one thousand: the human eye not only is variable inits resolution, it is highly variably in its resolution. Despite holdinggreat potential for improving display quality while simultaneouslyreducing the computational load, this variable resolution nature of theeye has for the most part not been exploited by any of our nowobliquities video technology. There is a simple reason for this: up tonow it has been very hard to do so. You can't record variable resolutionvideo, you don't know where the end-viewers are going to look. You can'treduce the cost of a display for the same reason. And all of our videoinfrastructure is based on constant resolution. All of our technologyfor rendering real-time 3D images only knows how to do so for (what wethink of as) constant resolution pixels.

The technology described here changes all that. An eye mounted display,such as a contact lens based display, one for each eye, has its physicaloutput pixels always where the eye is looking—so the pixels can bemanufactured of variable size, exactly matching the variable size of theeye's input pixels. A new rendering technology allows all the enormousadvances in 3D graphics rendering architectures to work directly invariable resolution pixel space—where more than an order of magnitudeless pixels have to be rendered.

An eye-mounted display, such as a contact-lens display, can be designedsuch that it can produce light from pixels that have been crafted to bein a space close to one-to-one with the location and size of the centerportions of the retinal receptor fields of the human eye. Because thisis a variable resolution space, only the position-based spatialfrequencies that the retina can perceive need to be rendered—but oneneeds a renderer that can advantageously do so. Later we will describehow the traditional computer graphics pipeline of points in threedimensional space being projected into uniform planer two dimensionalscreen space for rendering can be extended to projecting into a twodimensional but non-uniform non-planer screen rendering space that doesmatch the variable resolution and spherically curved surface of thehuman eye's retinal receptor fields.

Although the human eye has long since known to have a spherical imagingsurface, most past analyses of the eye have used the more familiarmathematical planer projection model employed by most man-made cameras,projectors, and computer rendering. Here new mathematical techniques aredeveloped to not only to understand imaging on a spherical surface, butalso how to understand what it means when the pixels on the sphericalsurface are extremely variable in resolution. Now the concept ofvariable perceptual resolution can be precisely defined and modeled.This technique can be reversed and used to develop variable resolutionpixel mappings with desired properties. A new mapping similar to that ofspatially variant resolution follows when the constraint of localorientation preservation is applied; this “locally uniform resolution”mapping matches much of the variable resolution of the human eye. It isalso used to create a tiled array of very small projectors, small enoughto fit within a contact lens, that only have as many pixels as the humaneye does.

A standard question is how a display device within a contact lens cangenerate images that will come into focus on the surface of the retina?One obviously cannot focus on images that are directly on top of one'scornea! The details of the types of optical wavefronts that can benormally focused on the retina, and how to generate them from within acontact lens are developed. The generation of the proper wavefrontsinvolves a large number of small projectors, each containing severalthousand pixels. How all these small projectors and their structuralsupports can be fabricated out of just four separate injection moldedparts (or their equitant) is described. Techniques to allow computergraphics pre-distortion of the rendered images reduce or eliminatedistortions due to the limits of fabrication and assembly techniqueswill be described.

FIG. 1 shows the solution in a general context. Element 110 is any formof electronic device that desires to have a visual output. Element 120consists of all of the elements of an eye mounted display system thatare required beyond the physical display device mounted on or in the eyeitself. Element 130 is the physical display device that is mounted on orin the eye itself. Element 140 is the human viewer.

FIG. 2 is a logical diagram of a complete eye mounted display systemelement 210 along with the human viewer 290. The four most importantsub-elements of the eye mounted display system element 210 are shownwithin it. Element 230, the scaler, is the subcomponent that convertseither video streams or GPU instructions element 220 into a frame bufferof variable resolution samples, which are then converted into outputpixels for the eye mounted display element 260. The position andorientation of the head of the human viewer element n is tracked by thehead tracker element 240. The position and orientation of the eye of thehuman viewer element 290 is tracked by the eye tracker element 250. Theeye mounted display element 260 produces a visual image element 280 thatcan be perceived by the human viewer element 290. Audio output element270 is also produced for consumption by the human viewer element 290.

FIG. 3 shows an eye mounted display system element 350 in the context ofa traditional computer element 310. The computer has input deviceselement 320, and output devices element 330, and an image generatorelement 340 (commonly a graphics card, or graphics chip, or portion of achip dedicated to graphics). The standard video output of the imagegenerator can be connected to the eye mounted display system element 350for display to the human viewer. Alternately, the eye mounted displaysystem element 350 can also serve as the image generator directly.

FIG. 4 shows a block diagram of a scaler element 405, which was element230 of FIG. 2. The scaler can accept video input element 410, andoptionally also accept graphics GPU input element 415. The scaler maycontain a higher order surface tessellator element 425. It may alsocontain a pixel shader element 430. The scaler contains a video outputconvolver element 435. Video frames, textures, and other large dataelements are preferably stored in an external bank of DRAM element 420.To speed access to this external data, preferably several caches areprovided. Element 440 is a video input cache. Element 445 is a texturecache. Element 450 is a sample buffer cache. Element 455 is a cache ofsamples waiting to be convolved for video output. There are two(logical) video outputs, element 460 for the left eye, and element 465for the right eye. They are “logical” because the scaler may not beconnected directly to the eye mounted displays, but indirectly via otherelements. Preferentially, the scaler element 405 is contained within asingle chip.

II. Anatomical and Visual Definitions

This section defines several specialized anatomical and visual termsrelated to the human eye. They are precisely defined here becauseunfortunately their usage is not consistent within the literature.

II.A. Terminology

We have both left and right eyes. Internally they are not the same, butmirror images of each other (e.g., as is true for our left and righthands, etc.). In order to talk about anatomical features of the eye, theliterature has adopted a terminology that is left/right eye agnostic.The term nasal means “in the direction of the nose,” and due to themirror symmetry, can describe the location of features of either eye.The opposite of nasal is the term temporal: “in the direction of thetemple.” While the eyes have are the same, not mirrored, in the verticaldirection, still in the literature the terms superior and inferior areused for the directions up and down. To unambiguously define what ismeant by front and back, the term anterior and posterior are defined tomean the front (where the light comes in the cornea) and rear (where theoptic nerve leaves) of the eye, respectively.

Because the optics of the eye inverts the image on the retina (as withmost man-made cameras), a location in visual space will correspond to aninverted (both horizontally and vertically) location on the innersurface of the eye (the retina). This means that the location of ananatomical feature on the retina is reversed depending upon whether oneis asking for its location on the surface of the retina (as one islooking at the eye from behind the eye), or the location in visualspace. A good example of all this is the blind spot. We have one in eacheye. Looking out into the world, visually the blind spot of the righteye appears to the right of center (temporal), but physically itslocation on the retina is inverted, and thus is located to the left ofcenter (nasal). The situation for the left eye is reversed, except thewords temporal and nasal don't change.

Due to the eyes optics, the visual image isn't just inverted, but alsosomewhat magnified onto the retina. Thus visual eccentricity (the anglethat a point in space makes relative to the visual axis) isn't exactlythe same as retinal eccentricity (the retinal angle made between (theray from the center of the retinal sphere to the center of the fovea)and (the ray from the center of the retinal sphere and the point on thesurface of the retina that a point in space project to through theoptics of the eye)). Because of this, it is important when locating ananatomical feature on the surface of the retina by describing itseccentricity, it is important to specify whether visual eccentricity orretinal eccentricity is being used, even when locating anatomicalfeatures relative to the physical retina itself. Many times in theliterature distance along the surface of the retina will not be given interms of either version of eccentricity, but in absolute distance units(generally millimeters: mm). Assuming a standard radius for the spherethat is the eye, such distances can be easily converted into retinaleccentricity. A problem is that many times the reference doesn't specifythe physical radius of the particular eye in question. And even whensome information is given, many times it is specified as the diameter ofthe eye. This is slightly ambiguous, as technically by diameter theymean axial length of the eye, which is not twice the radius of thespherical portion, but the distance from the front most bulge of thefront of the center of the cornea, to the back of the eye at theposterior pole. The axial length is usually a bit longer than thespherical diameter.

Also, while the size of a circular anatomical feature on the retinacentered on the fovea may be specified by a visual angular diameter,many times it is more important to think of it in terms of a maximumvisual eccentricity from the center of the fovea, e.g. half the value ofthe diameter. To make this point, usually both visual angles will bespecified.

Four figures depicting portions of the anatomy of the eye from the priorart are present first to give some context.

FIG. 5 shows a two dimensional horizontal cross section 500 through thethree dimensional human eye. This cross section 500 shows many of theanatomical and optical features of the human eye that are relevant tothe description of the invention. (Note: because the centers of thefovea 598 and the optic nerve 540 and optic disk 538 do not lie onexactly the same horizontal plane (more on this in a later section), thetwo dimensional horizontal cross section 500 is a simplification of thereal anatomy. However, this simplification is standard practice in mostof the literature; and so the slight inaccuracy usually does not have tobe explicitly called out. It is mentioned here because of the highlytight match there must be between the eye coupled display and the realhuman eye.)

To simplify this description, optical indices of refraction of variousgases, liquids, and solids will be stated for a single frequency(generally near the green visible optical frequency) rather than morecorrectly a specific function of optical frequency.

The outer shell of the eye is opaque white surface called the sclera506; only at a small portion in the front of the eye is the sclera 506replaced by the clear cellular cornea 650 (see FIG. 6).

FIG. 8 shows a two dimensional vertical cross section through the threedimensional human eye. The upper eye-lid 820 and the hairs attached to ϕthe upper eyelashes 830, along with the lower eye-lid 850 and lowereye-lashes 840, covers the entire eye during eye blinks, andredistribute the tear fluid 630 over the cellular cornea front surface660 (see FIG. 6). Not always noticed is that when one looks down, theupper eye-lid 820 moves down to cover the exposed sclera 506 almost downto the cellular cornea 508, colloquially the eyes are “hooded”.

The cellular cornea 650 is a fairly clear cell tissue volume whose shapeallows it to perform the function of a lens in an optical system. Itsshape is approximately that of a section of an ellipsoid; in many casesa more complex mathematical model of the shape is needed, and ultimatelymust be specific to a particular eye of a particular individual. Thethickness near the center of the cellular cornea 650 is nominally 0.58Millimeters.

FIG. 6 is a zoom into more detail of the cornea. The tissue at the frontsurface element 660 of the cellular cornea 650 is not optically smooth;a layer of tear fluid 630 fills in and covers these cellular cornealfront surface 660 imperfections. Thus the front surface 620 of this tearfluid layer 630 presents an optically smooth front surface to thephysical environment 910 (from FIG. 9); the combination of the cellularcornea 650 and the tear fluid layer 630 forms the physical and opticalelement called the cornea 514. While the physical environment 910 couldbe water or other liquids, gasses, or solids, for the purposes of thispatent it will be assumed that the physical environment 910 is comprisedof normal atmosphere at sea level pressures, so another name for 910 is“air”.

The optical index of refraction of the cornea 514 (at the nominalwavelength) is approximately 1.376, significantly different from that ofthe air (e.g., the physical environment 910) at an optical index of1.01, causing a significant change in the shape of the light wavefrontsas they pass from the physical environment 910 through the cornea 514.Viewing the human eye as an optical system, the cornea 514 providesnearly two-thirds of the wavefront shape changing, or “optical power” ofthe system. Momentarily switching to the ray model of light propagation,the cornea 514 will cause a significant bending of light rays as theypass through.

Behind the cornea 514 lies the anterior chamber 516, whose boarders aredefined by the surrounding anatomical tissues. This chamber is filledwith a fluid: the aqueous humor 518. The optical index of refraction ofthe aqueous humor fluid 518 is very similar to that of the cornea 514,so there is very little change in the shape of the light wavefronts asthey pass through the boundary of these two elements.

The next anatomical feature that can include or exclude portions ofwavefronts of light from perpetrating deeper into the eye is the iris520. The hole in the iris is the physical pupil 522. The size of thishole can be changed by the sphincter and dilator muscles in the iris520; such changes are described as the iris 520 dilating. The shape ofthe physical pupil 522 is slightly elliptical (rather than a perfectcircle), the center of the physical pupil 522 usually is offset from theoptical center of the cornea 514; the center may even change atdifferent dilations of the iris 520.

The iris 520 lies on top of the lens 524. This lens 524 has a variableoptical index of refraction; with higher indices towards its center. Theoptical power, or amount of ability to change the shape of wavefronts oflight passing through the lens 524 is not fixed; the zonules muscles 526can cause the lens to flatten, and thus have less optical power; orloosen, causing the lens the bulge and thus have greater optical power.This is how the human eye accommodates to focusing on objects atdifferent distances away. In wavefront terms, point source objectsfurther away have larger radius to their spherical wavefronts, and thusneed less modification in order to come into focus in the eye. The lens524 provides the remainder of the modifications to the opticalwavefronts passing through the eye; its variable shape means that it hasa varying optical power. Because the iris 520 lies on top of the lens524, when the lens 524 changes focus by expanding or contracting, theposition of the iris 520 and thus also the physical pupil 522 will movetowards or away from the cornea 514.

It is important to point out that this particular feature of the humaneye is slowly lost in middle age; by the late forties generally the lens524 no longer has the ability to change in shape, and thus the human eyeno longer has the ability to change its depth of focus. This is calledpresbyopia; present solutions to this are separate reading from distantglasses, or bifocals, trifocals, etc. In some cases replacing the lens524 with a manmade lens appears to restore much of the focus range ofthe younger eye. However, as will be discussed later, there are otherways to address the issue.

Behind the lens 524 lies the posterior chamber 528, whose boarders aredefined by the surrounding anatomical tissues. This chamber is filledwith a gel: the vitreous humor 530. In recent years it has been foundthat vitreous humor 530 is comprised not just of a simple gel, but alsocontains many microscopic support structures, such as cytoskeletons. Theoptical index of refraction of the rear of the lens 524 and the vitreoushumor 530 gel are different; this difference is included in themodifications to the shape of input wavefronts of light to the lens 524to the shape of the output wavefronts of light.

The inside surface lining of the eye is comprised of various thin layersof neural cells that together form a truncated spherical shell of suchcells that together are called the retina 534. It is shown in moredetail in FIG. 7. The highest resolution portion of the retina is thefovea 598. The edge of the spherical truncation that forms the outerextent of the retina within the eye is an edge called the ora serrata599. The anterior surface of the shell is bounded by the transition fromthe vitreous humor 530 to the retina; the rear of this thin shellbounded by the posterior surface of the pigment epithelium. The frontsurface of the shell is commonly defined as the retinal surface 730.However, when treating the retina as a photo sensitive surface, the sameterm “retinal surface” also commonly refers to a different surface: asub-layer within the particular layer within the thin neural layerswhere photons are actually captured: the photo retinal surface element720. In this document the term “retinal surface” will always refer tothe photo retinal surface 720.

II.B. Retinal Definitions

The Retina is a Subset of a Sphere

Definition of term: retina

Definition of term: retinal surface

Definition of term: point on the retinal surface

The retina consists of several thin layers of neurons on the innersurface of the eye. It covers approximately 65% of the inner surface,from the rear forward. This portion of the eye is nearly spherical. Thephrase retinal surface will be used when it is important to emphasizethat the retina is a thin non-planer surface. The phrase a point on theretinal surface will be used to indicate that a particular location onthe retina is meant; the fact that the retina, though thin, actuallyconsists of several relatively well defined layers is below this levelof abstraction.

Definition of term: retinal sphere

The retinal sphere is the best fit of a sphere to the retinal portion ofthe curved surface inner surface of the eye.

Definition of term: center of the retinal sphere

The center of the retinal sphere is the point located at the center ofthe retinal sphere.

Definition of term: retinal radius

The retinal radius is the radius of the retinal sphere. See thedefinition of the axial length for how these two terms differ. In thisdocument, we will assume a default retinal radius of 12 mm. It is calledthe retinal radius rather than the eye radius because it is the innerradius of the non-zero thickness spherical shell of the eye that is ofinterest, not the radius of the outer shell (the sclera).

Definition of term: corneal apex

The corneal apex is the front most point on the front surface of thecornea; typically this is also the point through which the cornealoptical axis passes.

Definition of term: axial length

The axial length of the eye is the distance measured from the cornealapex to the back of the eye at the posterior pole (thus this distance ismeasured along the corneal optical axis). The axial length is usually abit longer than the diameter (twice the retinal radius) of the (mostly)spherical portion of the eye. Statistically, human eyes have an averageaxial length of 24 mm, but individually vary in size from 20 to 30 mm.The distribution of this variance is approximately Gaussian about 24 mmwith a standard deviation of ±1 mm. Caution: when referring to the“diameter” of the (or an) eye, in the literature many times what ismeant is the axial length of the eye, not twice the retinal radius.Convention: while 24 mm is usually used as the “standard” or defaultaxial length of the eye, in this document we use 12 mm as the retinalradius, and the detailed model has an axial length of 23.94 mm.

Definition of term: ora serrata

On the inner surface of the eye, the edge where the retina ends iscalled to the ora serrata. It extends all the way around the inner frontsurface of the eye, but is not a line of constant retinal eccentricity.Instead its location varies in visual eccentricity from as little as 60°to as much as 105°. The exact shape is not well documented, but thegross details are. In the nasal direction the maximum visible visualeccentricity is close to 105°, but at the extreme, all you see is theside of your nose. In the superior and temporal directions, the maximumvisible visual eccentricity is around 60°. In the inferior direction themaximum visible visual eccentricity is around 65°. There is considerableindividual variation. The reason that each eye can see more of the worldtowards the nose is that that is where the stereo overlap between thetwo eyes lies.

Definition of term: posterior pole

The intersection of the optical axis of the cornea with the retinalsurface is called the posterior pole. It is the center of the retina, asfar as the axis of symmetry of the cornea (and thus the gross eye) isconcerned. However, for most purposes the center of the retina isinstead defined as the center of the fovea, which is located on theretinal surface, but in units of visual angle 2° inferior and 5°temporal from the posterior pole.

Optical Elements of the Eye

Definition of term: cornea

The cornea is the front most transparent optical element of the eye. Itis responsible for the first two-thirds of the optical power of the eye,most of the remainder is provided by the lens of the eye.

Definition of term: lens

Definition of term: crystalline lens

The lens of the eye is also known as the crystalline lens. Usage of thisphrase is considered mostly archaic in modern American English, but theusage is current in British English. In this document, sometimes thephrase crystalline lens will be used to avoid ambiguity when otherlenses are present.

Definition of term: accommodation

Definition of term: accommodation mechanism

The process of the eye dynamically changing the optical power of thelens to bring a point of visual attention into focus is calledaccommodation. accommodation is normally driven by vergence angle of thetwo eyes. That is, when the two eyes change their orientation such thatthe visual axis of the two eyes intersect at a point at a certaindistance from the eyes, that is a very strong indication that theoptical power of the two lenses should be changed so as to bring objectsnear that point into focus. In such a situation it is said that there isa change in the accommodation of the eyes, from wherever their previousfocus was, to the current desired focus. The overall process is calledthe accommodation mechanism. The accommodation mechanism is not normallyunder conscious control; normally the vergence angle directly drives thefocus of the eyes. This is a problem for stereo displays, as most cannotdynamically change the distance of optical focus of the display.

The Optic Nerve and the Optic Disc

Definition of term: optic nerve

Definition of term: lateral geniculate nucleus

Definition of term: LGN

Ganglion cells are located through the retina, with their dendritic endmostly placed locally to where their inputs from other retinal cellsare. However, the other end of all ganglion cells (the axons) all headfrom there across the retina toward the same spot: the optic disc. Theoptic disc is a hole in the retina where all these extended lengthganglion cells can pass through. When all these “nerve fibers” cometogether, they form the optic nerve. All communication from the eye tothe rest of the brain is via the nerve fibers bundled into the opticnerve. From the back of the eye, most, but not all, of the nerve fibershead into the portion of the brain known as the lateral geniculatenucleus (LGN). There are actually two LGNs, a left and a right one.

Definition of term: optic disc

There is a hole in the surface of the retina where a large bundle ofnerves from the retina become the optic nerve and exit through the backof the eye. This hole is called the optic disc, though its shape iselliptical: its size is approximately 1.5 mm horizontally and 2 mmvertically. Relative to the posterior pole (the end of the opticalaxis), on the surface of the retina the optic disc is centeredvertically (0°), and located 10° nasal. Relative to the center of thefovea, the optic disc is located 2° superior, and 15° nasal.

Definition of term: blind spot

Visually, the lack of photoreceptors (rods or cones) caused within theoptic disc results in what is called the blind spot. In visual space,relative to the visual axis, it is located 2° inferior, and 15°temporal.

The Macula

Definition of term: macula

Definition of term: macula lutea

The macula, also known as the macula lutea, is a circular disk ofyellowish pigment on the retina centered on the fovea. The thickness ofthe macula diminishes with distance from the center of the fovea, butsome of the same pigment that makes up the macula is found throughoutthe rest of the retina. Thus determining exactly where the macula endsis a subjective anatomy call; different sources express its diameter invisual space as anywhere from 5° to 20°, corresponding to a maximumvisual eccentricity of between 2.5° and 10°. In addition to this, it isknown the extent of the macula, as well as the peak thickness, issubject to a fair amount of individual variation.

One presumed function of the macula is to greatly reduce the amount ofshort wave length light (blue through ultraviolet) that reaches thecentral retina (that hasn't already been absorbed by the cornea andlens).

The Fovea

Definition of term: anatomical fovea

Definition of term: fovea centralis

The term “fovea” unfortunately has two different definitions. One isdefined by anatomical features, which we will always refer to as theanatomical fovea. This anatomical definition, also known as the foveacentralis, is a circular area of the retina 5° of visual angle indiameter (1.5 mm), or a radius of 2.5° of visual eccentricity, withinwhich most of the retinal neural layers are absent. This absence allowsfor best optical imaging quality to the photoreceptors present. Thelocation of the anatomical fovea on the retina, relative to theposterior pole, is 2° superior and 5° nasal (there is some individualvariation). In terms of anatomical features, the definition of the edgeof the anatomical fovea is the location where the layers of retinalcells achieves their maximum density (thickness). From a resolutionpoint of view, the term “fovea” has a different definition as a smallerregion of the retina.

Definition of term: fovea

From a resolution point of view the term “fovea” indicates a region ofvisual space where the eye has its maximum resolution. In this document,we use the plain term fovea to indicate this visual definition.Specifically, the visual fovea is defined as a circular area of theretina 2° of visual angle in diameter (0.3 mm), or a radius of 1° ofvisual eccentricity, with the same center on the retina as theanatomical fovea

Definition of term: center of the fovea

By the phrase the center of the fovea, we will always mean the point atthe center of the highest resolution (smallest cones) portion of theanatomical fovea. Anatomically, this is the geometric center of thefoveal maximum cone density zone.

Anatomically Defined Sub-Regions of the Fovea

Definition of term: foveal avascular zone

The foveal avascular zone is a circular sub-region of the anatomicalfovea about 1.4° of visual angle across (0.4 mm), or a circle of 0.7° ofvisual eccentricity, anatomically defined as where even blood vesselsare absent.

Definition of term: foveal rod-free zone

The foveal rod-free zone is a circular sub-region of the anatomicalfovea about 1° of visual angle across (0.35 mm), or a circle of 0.5° ofvisual eccentricity, anatomically defined as where only conephotoreceptors are present, not rods.

Definition of term: foveal blue-cone-free zone

The foveal blue-cone-free zone is a circular sub-region of theanatomical fovea about 0.35° of visual angle across (0.1 mm), or acircle of 0.17° of visual eccentricity, anatomically defined as where noblue cones are present, only red cones and green cones.

Definition of term: foveola

The term foveola (“little fovea”) atomically refers to a central portionof the anatomical fovea where no ganglion cell layer exists. But someauthors define it differently, it can mean any one of the terms fovealavascular zone, foveal rod-free zone, or the foveal blue-cone-free zone.We will avoid the ambiguity by not using this term here, but the morespecific ones instead.

Definition of term: foveal maximum cone density zone

Within a small very most central portion of the anatomical fovea,anatomically lies a circular sub-region of the anatomical fovea wherethe density of cones is at its maximum (for that individual). This is aregion of approximately constant size cones that have the smallest size(highest resolution) found on the retina. This foveal maximum conedensity zone is only four minutes of visual arc (1°/15) across (0.02mm), or a circle of two minutes of visual eccentricity. This is wellwithin the foveal avascular zone, foveal rod-free zone, and the fovealblue-cone-free zone, so in the foveal maximum cone density zone thereare only red cones and green cones, no blood vessels, and no rods. Theregion is so small that it may contain less than 50 cones.

Definition of term: foveal maximum cone density

This is the per-individual peak cone density of their fovea, and it canvary by individual from as little as 150,000 cones/mm to as much as350,000 cones/mm.

Definition of term: periphery

The rest of the retina outside the anatomical fovea is referred to asthe periphery. It extends from the outside edge of the anatomical fovea,at 2.5° of visual eccentricity, all the way to the ora serrata, at 65°to 105° of visual eccentricity. Caution: when the visual fovea is thecontext, the term periphery can instead refer to the region outside thissmaller region, e.g. extending from 1° of eccentricity out. Which ismeant to be inferred from context.

II.C. Axes: Classical Optical Vs. the Human Eye

Optical Axes of Classical Optical Systems

In classical optics, the concept of “the optical axis” is well defined.Most classical optical systems are circularly symmetric (centered),making the optical axis easy to find: it is the axis of symmetry. Evenmost “off-axis” classical systems are really just a portion of a largersystem that would have circular symmetry, and again the optical axis isself-evident. Quite often there is just one optical axis, which is whyone can talk about “the” optical axis.

Optical Axes of the Human Eye

The human eye, however, is most certainly not a classical opticalsystem, except in quite abbreviated form. The individual opticalelements of the eye are mostly circularly symmetrical; the problem isthat the individual optical axes are not aligned with each other. Muchof this comes about because the axis of symmetry of the main imagesensing component of the eye, the retina, is tilted by 5° relative tothe main optical bending component of the eye, the cornea. Because ofthis, the axis of the pupil is offset from the corneal axis, and theaxis of the lens is tilted relative to the cornea. The result is adecentered optical system, and concepts like the “optical axis” areill-defined. To further complicate things, the center of rotation of theeye is not on any of the other axes, and its location actually changesduring rotation.

Traditionally these issues are addressed by having different levels ofapproximation to optical models of the human eye. More recently some ofthe issues have been addressed by making changes or redefinitions interminology (visual axis vs. line of sight).

Optical Models the Human Eye

All optical models of the human eye must be “simplified” models at somelevel, for example, very few contain an optical model of everyphotosensitive cone cell (let alone rod cell) in the eye. But even asimplified model can be quite useful for a given objective: fittingspectacles, or just understanding the image forming process. Some of thesimplest sort of optical models are called schematic eyes, which arefurther divided into simple paraxial schematic eyes, and the morecomplex wide angle schematic eyes.

Definition of term: schematic eye

A schematic eye is an approximate model of the optics of the human eye,which has been simplified into a small number of classical opticalcomponents.

Definition of term: paraxial schematic eye

Definition of term: paraxial region

A paraxial schematic eye is a schematic eye which is explicitlyrestricted to be valid only within the paraxial region: optical anglesless than 1°, where sin [x]≈x.

Definition of term: wide angle schematic eye

A wide angle schematic eye is a schematic eye which is explicitly validat nearly all angles, well beyond the paraxial region. In the human eye,this can extend to visual eccentricities past 90° (a field of view of180°), up to 105° or more.

While any paper describing a particular optical model of the human eyecan be said to define a schematic eye, there are a few well knownhistoric schematic eyes that are commonly used as references. Most ofthese are paraxial schematic eyes, but a few are wide angle schematiceyes. Unfortunately from an axis point of view, pretty much allschematic eyes have a single optical axis, thus they are not very wellsuited for simulation of eyes with fovea's located properly at 5° offthe corneal optical axis. One exception is [Deering, M. 2005. A PhotonAccurate Model of the Human Eye. ACM Transactions on Graphics, 24, 3,649-658], and that model will be used here.

The Axis of the Eye

Definition of term: optical axis of the cornea

The cornea is easily accessible from outside the eye, and from its shapeits function as an optical element can be understood, including itsprinciple optical axis. This axis is defined as a line through thecenter of the cornea, normal to the surface of the cornea there. Wherethe optical axis of the cornea hits the surface of the retina is calledthe posterior pole. Unfortunately the rest of the optical elements ofthe eye, including the pupil, the lens, and the spherical imaging plane(the retina), all have distinct, and different from each other, axis.Thus there is no single “optical axis of the eye.” The term still arisesin the literature, and sometimes is synonymous with optical axis of thecornea, and other times synonymous with the visual axis. We will avoidambiguity by not defining or using the phrase “optical axis of the eye.”

Definition of term: optical axis of the pupil

The optical axis of the pupil is defined to be a line through the centerof “the hole in the iris.” The orientation of the line is normal to theorientation of the inside edge of the iris. But this orientation isn'tvery well defined, and the lens can actually bulge through the iris. Thecenter of the hole in the iris is not aligned with the optical axis ofthe cornea, it is generally decentered from it by 0.25 to 0.5 mm, andvaries by individual. This decentering is required due to the resolutioncenter of the retina (the center of the fovea) being 5° offset from theintersection of the optical axis of the cornea and the retina (theposterior pole). And not only that, but as the iris dilates (openslarger) the center of the hole can actually shift by 0.1 mm or more.Furthermore, the actual shape of the hole is not a perfect circle, butslightly elliptical (^(˜)6%). In most all the published schematic eyes,including wide angle schematic eyes, the optical axis of the pupil (andthe optical axis of the lens) are assumed to be the same as the opticalaxis of the cornea. But for accurate modeling, the appropriate offsetmust be included.

Definition of term: optical axis of the lens

The optical axis of the lens is also decentered and tilted from that ofthe optical axis of the cornea, though by how much is still somewhatspeculative. From an accurate optical modeling point of view, thedecentering of the pupil is more important to include than decenteringand/or tilting the lens.

Definition of term: visual axis

Definition of term: line of sight

From any given direction d towards the eye, a pencil of rays, all withthe same direction, but offset in space, can hit the front surface ofthe cornea. After bending by the cornea's optical function, the pencilof rays will continue deeper into the eye, and many will hit the planeof the iris. However, of the original pencil of rays, only a subset willactually pass through the hole in the iris (the pupil), and continueeven deeper into the eye. Label this subset pencil of rays all with thesame direction d in object space that will end up passing through thepupil pd. The rays within any such pencil will still be offset in spacefrom one another, generally in an elliptical cross section (as cut by aplane tangent to the center of the cornea). One can chose a singleprinciple ray of each pencil ppd by choosing the ray at the center ofthe pencil. Then, over all directions d, (generally) only one of theseprinciple rays will, after being further bent by the eye's lens, hit thesurface of the retina at the exact center of the fovea. This one raydefines the visual axis.

Many times a different older definition of the visual axis is used: theray in the visual world that ends up passing through the first nodalpoint of the eye's optics. Those who use the older definition now usethe phrase line of sight to mean the same thing as the newer definitionof the visual axis. But because even the line of sight has multipleolder meanings, in this document the visual axis will always be used inits new form instead.

Definition of term: Eye Point

Definition of term: ViewPoint

The simple computer graphics view model is equivalent to pin-hole lensoptics. That is, the only rays in object space that will be consideredfor making up the image plane are rays from any directions but that allpass through the same point: the Eye Point, or ViewPoint, which is thesame location as the (infinitively small) pin-hole.

In real optical systems with lenses and non-infinitesimal entrancepupils, the situation is more complex, but the EyePoint is known to belocated at the first nodal point of the optical system. Unfortunately,the last is true only for optical systems with rotational symmetry abouta single shared optical axis. While this characterizes most man-madeoptical systems, the decentered and tilted elements of the optics of thehuman eye makes this not true there.

In the optical system of the human eye, the EyePoint will be locatedsomewhere on the line defined by the visual axis. Referring to theconstruction of the visual axis, in theory all the principle rays ppd,if left to continue un-deflected by the eye's optics, would eventuallyintersect at a single point, and this would define the EyePoint. Inpractice, all the rays won't quite intersect at a single point, so thebest one can do is chose the center of the narrowest waist in theenvelope of such rays as they “almost intersect.” These other locationsin space within the envelope at its narrowest represent a more generalconcept of a “region” of EyePoints, and they are used in more complexcomputer graphics rendering models to correctly simulate depth of fieldeffects.

Note that technically the EyePoint can move in and out slightly alongthe visual axis as accommodation changes. This is because as the eye'slens bulges or flattens to change focus, the location of narrowest pointin the envelope are the rays cross will change as the optical powerchanges. One can also see from the definition of the EyePoint that ifthe center of the hole in the iris shifts laterally as the iris opens orcloses, then both the visual axis and the EyePoint will shift laterallysome as well.

Definition of term: center of the exit pupil of the eye

In an optical system with rotational optical symmetry, there is atheoretical single point that all the rays exiting from the opticalsystem (e.g., those heading towards the imaging surface) will seem tohave come from. In such classical optical systems, this is the secondnodal point of the optical system. But once again, the human eye doesn'tplay by the same rules.

What we want to know is where, from points on the surface of theretina's point of view, do all the rays coming to the surface of theretina seem to come from? Again, the answer is a generalization that isbased on finding the narrowest waist in the bundles of rays that emergefrom the lens on the inside of the eye. This point is known as center ofthe exit pupil of the eye.

Definition of term: rotational center of the eye

The human eye center of rotation is not fixed; it shifts up or down orleft or right by a few hundred microns over a ±20° rotation fromstraight ahead. The “standard” non-moving (average) center is given as apoint on a horizontal plane through the eye, 13 mm behind the cornealapex, and 0.5 mm nasal to the visual axis.

Definition of term: retinal visual axis

The retinal visual axis is defined as ray from the center of the retinalsphere to the center of the fovea on the surface of the retina.

Definition of term: visual angle

The visual angle between two points in space is the angle between theray from the Eye Point to the first point and the ray from the Eye Pointto the second point. The visual angle between two rays in space with acommon origin at the Eye Point is the angle between the two rays. Thevisual angle between a ray in space from the Eye Point and a point inspace is the angle between the ray and the ray from the Eye Point to thepoint in space. Note that the visual angle between two points in spacewill not be exactly the same as the retinal angle of the two points onthe surface of the retina that the optics of the eye project each pointin space to.

Definition of term: visual eccentricity

The visual eccentricity of a point in space is the visual angle betweenthat point and the visual axis. The visual eccentricity of a ray inspace with its origin at the EyePoint is the angle between that ray andthe visual axis. Note that the visual eccentricity of a point in spaceis not exactly the same as the retinal eccentricity of the point on thesurface of the retina that the optics of the eye project the point inspace to.

Definition of term: retinal angle

The retinal angle between two points on the surface of the retina is theangle between the ray from the center of the retinal sphere to the firstpoint and the ray from the center of the retinal sphere to the secondpoint. The retinal angle between two rays with a common origin of thecenter of the retinal sphere is the angle between them. The retinalangle between a ray from the center of the retinal sphere and a point onthe surface of the retina is the angle between the ray and the ray fromthe center of the retinal sphere and the point on the surface of theretina. Note that the retinal angle between two points on the surface ofthe retina will not be exactly the same as the visual angle between tworays that reverse of the optics of the eye would project each point onthe retina out to as rays in object space.

Definition of term: retinal eccentricity

The retinal eccentricity of a point on the surface of the retina is theretinal angle between that point and the retinal visual axis. Theretinal eccentricity of a ray with its origin at the center of theretinal sphere is the angle between that ray and the retinal visualaxis. Note that the retinal eccentricity of the point on the retina willnot be exactly the same that the visual eccentricity that the inverse ofthe optics of the eye project that point out to as a ray in objectspace.

Definition of term: retinal distance

The retinal distance is a distance between two points on the surface ofthe retina along the great circle connecting them on the retinal sphere,measured in millimeters. If you know what the retinal radius is, then aretinal distance can be converted to a retinal angle. This can beconverted to a visual angle, but only by a complex function of the eyemodel. When one of the two points on the retina is the center of thefovea, then the retinal distance, when the retinal radius is known, canbe converted to retinal eccentricity (and then the conversion to visualeccentricity is a bit easier, though still not straight forward). Whenthe distance between the two points is small relative to the retinalradius, then the great circle distance between the two points iseffectively the same as their Euclidean distance. A good example of whenthis approximation apples is in specifying the size and spacing ofretinal cones.

Definition of term: retinal exit pupil angle

The retinal exit pupil angle of a point on the retinal surface isdefined as the angle between the ray from the center of the exit pupilof the eye to the point with the retinal visual axis. This angle issimilar to the visual eccentricity, but like the retinal eccentricity,they are not related by a closed form expression. A common approximationis that tan [retinal exit pupil angle]=0.82·tan [visual eccentricity].

Definition of term: inter-pupillary distance

Simplistically, the inter-pupillary distance is just the distancebetween the (centers of) the two pupils of a viewer's two eyes. But thisdistance will change when the vergence angle between the two eye'schange. In practice, the inter-pupillary distance is measured when theviewer is looking straight ahead and is focused on a point a greatdistance away (a vergence angle near zero).

Here we will more rigorously define the inter-pupillary distance as thedistance between the rays of the visual axis of a viewer's left andright eyes, when the two rays are parallel and the eyes are lookingstraight ahead. Visually, this occurs when the viewer is looking at anobject at an infinite distance. While objects can optically be placed atthe equivalent distance to infinity, in practice distances of greaterthan two miles are indistinguishable, and even distance as little asthirty or forty feet are a good approximation.

One might think that certain anatomical measures would be provide thesame value, e.g. the distance between the center of the retinal sphereof the viewer's two eyes, or the distance between the rotational centerof the eye of the viewer's two eyes. However, due to the decenterednature of the optics of the eye, these anatomical measures are not quiteequivalent.

Another complication to consider is that historically theinter-pupillary distance many times has been considered the purelyhorizontal distance between the centers of the pupils of the eyes. Thatis, a measurement made parallel to the ground when the viewer's head isperfectly upright. Unfortunately human eyes are not at the same level,usually one will be a little higher than the other relative to theupright skull. This height difference can be significant enough that itmust be accurately measured. Along the same lines, the distance from theplane of symmetry of the skull to the left and right eyes also usuallydiffers, and can also be significant enough that it must be accuratelymeasured. What is ultimately required, rather than the historicinter-pupillary distance measurement, is the exact location relative toa fixed coordinate frame of the skull, is the 3D location of the leftand right EyePoints when the eyes are at rotational rest, and theorientation of the visual axis relative to the same skull coordinatesystem under the same conditions. Here inter-pupillary distance wasdefined in such a way that it measures the true distance between theinfinity directed visual axis of the viewer's two eyes, not just thehorizontal component.

While in the general population a viewer's eyes tend to both point tothe same location in space where attention is being directed, in peoplewith strabismus, or similar conditions, the two eyes are not coordinatedin the same way, and thus alternate means of measuring inter-pupillarydistance need to be used.

Definition of term: vergence angle

Definition of term: angle of vergence

Under normal stereo viewing conditions, a viewer will orientate theireyes such that the left and right visual axis will intersect (or comevery close to intersecting) at a point in space where attention is beingdirected. No such point exists in the special case of viewing atinfinity, but in all other cases such a point will exist somewhere inphysical space in front of the viewer. The definition of the vergenceangle, or angle of vergence, is the angle made between the left andright visual axis from this point of intersection.

While the vergence angle is often associated with the current distanceto the point of visual attention, the same vergence angle does notexactly correspond to the same distance in space, especially when thepoint towards the far left or right of the visual region of stereooverlap. However, to a first approximation, the vergence angle is a goodestimate of the distance to the current point of visual fixation, andthe focus system of the human eye, the accommodation mechanism, uses thecurrent vergence angle to drive the current focus (e.g., at whatdistance should objects currently be in focus). That is, if you know aviewer's inter-pupillary distance, and the current vergence angle, thenby simple trigonometry the distance to the current point of visualfixation is approximately:

$\begin{matrix}{{distance} = \frac{{inter}\text{-}{pupillaryDistance}}{\sin\lbrack{vergenceAngle}\rbrack}} & (1)\end{matrix}$

In such a way, dynamically knowing the vergence angle of a viewer'seyes, one thus also known with high probability what the current depthof focus of their eyes are. This is important in display applicationswhere the optical depth of field of the display must be dynamicallyadjusted to match what the human visual system is expecting.

For viewers with conditions like strabismus, vergence angle does notcarry the same information.

Common Deficiencies of the Eye

Definition of term: myopia

Definition of term: myopic

Definition of term: high-myopic

Definition of term: near-sightedness

Myopia, also called near-sightedness, occurs when the optical system ofthe eye causes distant objects to come into focus not on the surface ofthe retina, but at a location in front of it. A person with thiscondition will be able to bring objects relatively close to them intofocus, but will not be able to do so for objects past some distanceaway. Such a person is said to be myopic. A severely nearsighted personis described a high myopic. The cause of myopia is usually an elongationof the eye, making it more ellipsoidal than spherical.

Definition of term: hypermetropia

Definition of term: hypermetropic

Definition of term: far-sightedness

Hypermetropia, also called far-sightedness, occurs when the opticalsystem of the eye causes near objects to come into focus not on thesurface of the retina, but at a location in back of it. A person withthis condition will be able to accurately focus on objects relativelyfar away, but will not be able to do so for objects closer than somedistance. Such a person is said to be hypermetropic. The cause ofhypermetropia is usually a foreshortening of the eye, making it moreellipsoidal than spherical. hypermetropia is not to be confused withpresbyopia, a condition that naturally occurs with aging.

Definition of term: astigmatism

Definition of term: astigmatic

Astigmatism in general refers to an optical system whose main opticalelement does not possess circular symmetry. In the case of the eye, themain optical element is the cornea, and it normally is circularlysymmetric. However, in cases where the cornea is more ellipsoidal thancircular, circular symmetry no longer holds, and the eye is thenconsidered astigmatic, and such a person is considered to haveastigmatism. In such cases, the cornea no longer possess a singleoptical power, but a range of optical powers depending on theorientation of a stimulus to the eye. For the minority of cases in whichthe eye is astigmatic, but the fault is not the cornea (or entirelywithin the cornea), the problem then usually is the lens, and isgenerally an early indication of cataracts.

Definition of term: presbyopia

While the lens starts out in life as quite flexible and is easilychanged in shape by the appropriate muscles when different opticalpowers are required, later in life (towards the age of forty) too manyouter layers have been added to the lens and it starts to “harden”,becoming less and less flexible, and thus possessing a lesser and lesserrange of accommodation (focus). This (natural) condition is calledpresbyopia, it is why people generally require reading glasses later inlife, even if their normal distance vision still does not requirecorrection. For those who also require some form of optical correction,bifocals are one mechanism by which a single pair of glasses (or contactlens) can be support both distance and close up vision.

Definition of term: strabismus

Strabismus is a medical condition in which a person's two eyes do notalways change their orientation in tandem with each other. Such peoplegenerally do not have stereo vision.

II.D Important Retinal Neural Cell Types

While the cornea and the lens of the human eye are constructed frommodified skin cells, the light sensing and information processing cellsof the retina are all specialized type of neurons: the brain literallybegins inside the back of the eye. This section will define a few of themost important such cells that we will need to consider later.

Photoreceptors

Definition of term: photoreceptors

The first class of neural cells we will describe are photoreceptors,those neurons whose main purpose is to capture photons of light, turneach photon event into increments of electrical charge, and turn thesummed voltage over a time span into the release of neural transmittermolecules that act as inputs to other retinal neural cells.

Definition of term: cone cell

Definition of term: cone

Definition of term: red cone

Definition of term: green cone

Definition of term: blue cone

The cone cells, or just cones, are the daylight and indoor lightingsensitive photoreceptors of the eye. These photoreceptors neurons areactive a moderately low to very high light levels, and pass a temporallynormalized light level value to their output. The cone cells come threetypes with different spectral sensitivities to light: long wavelength,mid wavelength, and short wavelength, or now more commonly referred toas: red cones, green cones, and blue cones.

Definition of term: rod cell

Definition of term: rod

The rod cells, or just rods, are the nighttime and dark sensitivephotoreceptors of the eye. These neurons are active at extremely lowlight levels, and become inactive as the light level approaches amoderately low level. Unlike cones, the output of rod cells is nottemporally normalized; their outputs can always be read as a (timeaveraged) direct photon event count. The rod cells come in only onespectral sensitivity to light; they only “see” in black and white(shades of gray).

Horizontal Cells

Definition of term: horizontal cell

The horizontal cells connect to nearby cone cells. They appear to bothinput from and output to the cone cells they connect to. Their purposeappears to be to act as a contrast enhancer of the signal that the conecells put out. Different types of horizontal cells exist with differentpreferences as to which color(s) of cone cells they connect to and from(some also connect to rods). Some have proposed that the horizontalcells eventually contribute to the surround portion of the midgetganglion cell receptor field, but the evidence is not conclusive.

Bipolar Cells

Definition of term: bipolar cell

The bipolar cells take their input(s) from photoreceptors, and output toone specific ganglion cell. Bipolar cells can be classed in onedimension as either those that invert their input or those that do not.In other dimensions, there are several types of bipolar cells, but wewill be concerned with the subtypes that takes their input from a singlecone cell.

Definition of term: cone bipolar cell

The cone bipolar cells take input from a single cone cell. One typetakes it input from either a single red cone or green cone, anothertakes its input from a single blue cone. Each of these cone bipolar celltypes come in both inverting and non-inverting forms.

Amacrine Cells

Definition of term: amacrine cell

The amacrine cells connect with both bipolar cells and ganglion cells,as well as each other. There are many different specialized types ofamacrine cells. Currently it is thought that some of these are whatforms the surround portion input to the midget ganglion cell receptorfield out of the outputs of nearby cone bipolar cells.

Ganglion Cells

Definition of term: ganglion cell

The ganglion cells of the retina represent the final stage of retinalprocessing of visual information. So far all the other retinal celltypes take in and send out “analog” values. For input, they have takingin quanta of light (photoreceptors), electrical potential (gapjunctions), or quantities of neural transmitter molecules. These all areturned into electrical potential inside the cell, processed in some way,and then output, again in one (or more) of the same three ways. Ganglioncells are different. While they take their inputs in as neuraltransmitter molecules, their output is in the new form of pulses ofaction potentials for long distance signaling of effectively digitalinformation to the brain proper. Ganglion cells are very long; they haveone foot in the retina, but the other foot in the brain, usually at theLGN. The output of each ganglion cell heads from wherever it is locatedon the retina towards the exit out the back of the eye, the optic disc,where they become part of the optic nerve. The optic nerve heads intothe brain, where most (though not all) ganglion cells have their otherend terminate within the LGN.

Ganglion cells come in four different “sizes,” which vary in how largeof visual field they take in. We will only be interested in the smallestsize, midget ganglion cells, because they are the determiner ofresolution. Each of these come in an “OFF” and “ON” type, and arefurther differentiated by the color of the cones that they areindirectly connected to.

Definition of term: midget ganglion cell

The midget ganglion cells take input from cone bipolar cells, and, likemost ganglion cells, send their output to the LGN.

Definition of term: retinal receptor field

The receptor field of a given neuron in the retina can be defined as thesub region of visual space over which increments or decrements of lightcan cause the output of that particular neuron to change. Most retinalreceptor fields are fairly narrow in extent, their visual regions arebetween half a minute of arc to a few minutes of arc in visual anglediameter. While by the definition of receptor field, all retinal neuronshave a receptor field, for simplicity in this document the phraseretinal receptor field, without further identification of the cell typeinvolved, will refer to the receptor field of the last stage of highresolution processing in the retina, e.g. the midget ganglion cellreceptor field. The center portion of this receptor field will sometimesalso (loosely) be referred to as “the pixels of the eye.”

Definition of term: midget ganglion cell receptor field

Midget ganglion cells have a receptor field characterized by a smallcentral field and a larger surround field. It is important to note thatthe surround field includes the region of the center field as a subfield.

Up to about 6° of visual eccentricity, each midget ganglion cell centralreceptor field takes input from only one cone bipolar cell output, whichin turn is connected to only one cone cell (and therefore of only onecolor type). Between 6° and 12° of visual eccentricity, each midgetganglion cell central receptor field takes as input either one or twocone bipolar cell outputs. The proportion of input connections to onevs. two cone bipolar cells varies smoothly from mostly one just above 6°to mostly two just below 12°. Between 12° and 16°, the number of midgetganglion cell central receptor field inputs from different cone bipolarcell outputs smoothly varies from two to three in a similar manner. Thisprocess continues at increasing visual eccentricities, and by 45° ofvisual eccentricity, the number of pooled inputs is twenty one. How isthis overall function characterized? Analysis of anatomical midgetganglion cell density counts reveals that the number of midget ganglioncells present on the retina at a particular visual eccentricity isapproximately constant. This means that the implied mapping is close tothat of LocallyUniformResolution.

What about the surround portion of the midget ganglion cell receptorfield? When the center portion is just one cone bipolar cell, thesurround appears to be from just the inverted version of that cellagain, plus from the inverted version of the (approximately) six conebipolar cells whose input cone cells are the immediate neighbors of thecone cell input to the central cone bipolar cell. When the centerportion of the receptor field is from multiple cone bipolar cells, thenthe surround appears to be from a proportionately larger number of conebipolar cells with inputs from close neighboring cone cells. It ispresently thought that a single amacrine cell takes its input from themultiple (inverted) cone bipolar cell outputs, and the single amacrinecell output is connected to the input of a single midget ganglion cellin order to form the surround portion of the midget ganglion cellreceptor field.

Definition of term: midget ganglion cell processing

The processing of a midget ganglion cell is to subtract the value of itssurround field from its central field, or vice versa. Because the outputof midget ganglion cells is in the digital pulse code format (used bymost of the rest of the brain) which can only represent positivenumbers, midget ganglion cells whose central receptor field input comesfrom one or more cone bipolar cells of the “ON” type subtract the valueof their surround input from that of the center input, and clip if belowzero. E.g., they only send out pulses if the result of the subtractionis a positive number. Midget ganglion cells whose central receptor fieldinput comes from one or more cone bipolar cells of the “OFF” typesubtract the value of their central input from that of the surroundinput, and clip if below zero. E.g., they only send out pulses if theresult of the subtraction the other way would have been a negativenumber. Since each midget ganglion cell has a direct line to the brain(at the LGN) through its other end, that means that in principle each“pixel” of the retina takes two neural fibers in the optic nerve totransmit its value to the brain, because of the need to encode valuesinto only positive numbers. As the output of midget ganglion cells formthe majority of neurons in the optic nerve, this in essence means thatthe number of neural fibers in the optic nerve represents an approximatedouble count of the number of “pixels” actually sensed by the eye. Inactuality, this is only strictly true for the “ON” and “OFF” pairs ofmidget ganglion cells below 6° of visual eccentricity. Above this, thepairing is no longer exact, as the “ON” and “OFF” sets of midgetganglion cells separately choose when and how they connect to more thanone cone bipolar cell. In fact, the ratio of the density of “ON” midgetganglion cells to “OFF” midget ganglion cells in the periphery appearsto be about 1.7:1 (1.3:1 difference in linear resolution). Someperceptual experiments appear to confirm that this ratio effects humanperceptual resolution.

III. Mathematical Definitions

III.A Mathematical Conventions

Mathematical Conventions on Variable Names

Traditional mathematical notation heavily relays on the convention thatall variable names are a single character in length. They may have(multiple) subscripts or superscripts, be “primed”, “tildiaded”,boldfaced, etc., but there is only one base character. When amathematician runs out of letters in the alphabet, he just starts usingletters from another alphabet. When this isn't sufficient, subscriptingis added, even if that means subscripted subscripts. In computerscience, variable names can be arbitrarily long, but (usually) don'tinclude spaces between multiple words that make up a long name, insteadeither the “under-bar” “_” is used, or the first character of the eachname (not always including the first) is capitalize. In this document wewill generally use multiple character variable names, with the firstletter of a word uppercased convention. Occasionally, in relativelyinformal equations, phrases with spaces between words will be used forvariables, when there is little or no possibility of ambiguity.

Mathematical Conventions on Multiplication

Traditional mathematical notation for multiplication heavily relays onthe convention that all variable names are a single character in length.Because multiplication is so common, the traditional convention is thattwo variables right next to each other (e.g. usually with no spacebetween them) indicates an implicit multiplication.

But because we are not adhering to the single character variable nameconvention in this document, we always will have to include an explicitmultiplication operator symbol of some sort. For ordinary multiplication(e.g. of scalars, or multiplication of something else by scalars), wewill use the centered small dot “·” to indicate multiplication.(Computer science usually uses the asterisk “*” (or occasionally thecentered star: “*”) for multiplication, or even “mul”.) The centeredsmall dot is the normal mathematical notation for multiplication when anexplicit multiplication operator symbol is required.

A problem arises in multiplication of objects other than by scalars. Dotproducts and cross products of vectors can use their traditionaloperator symbols: ● and x. But for matrix multiplication, there is nostandard convention of what symbol should represent the matrixmultiplication operator when an explicit operator is required. Sometimesthe small dot is used, sometimes the cross product symbol is used. Sincewe have to make a choice here, the cross product symbol “x” will beused. While this is slightly less common than the choice of the smalldot, it reinforces the semantics that matrices are being multipliedhere, not scalars.

Another issue with matrix multiplication is the semantics of the orderof their composition under multiplication. With scalars, multiplicationis commutative: x·y·z=z·y·x, so in mathematics the order in whichscalars are multiplied is irrelevant. In computer science, because ofrounding, the order in which numbers are multiplied can be relevant, sothe default order of application is in general left to right, thoughoptimizing compilers are allowed to use the commutatively property tore-shuffle the application order in most circumstances. However matrixmultiplication is in general not commutative, so the order makes adifference. This is important because matrix multiplication is definedto operate right to left, not left to right. The reason for this is thatmatrices are considered transforms, which are considered mappings, whichhave associated mapping functions. So for example, assume that you havethree spaces: A, B, and C. Then assume that the transform between pointsin the first to points in the second is AtoB( ), and the transformbetween points in the second to points in the third is BtoC( ). Now thetransform between points in the first to points in the third is definedby the application of BtoC on the results of applying AtoB:BtoC(AtoB)→AtoC( ), or BtoC° AtoB→AtoC. This makes perfect sense whenthe transforms are viewed as functions. But the mathematical conventionis that even when transforms are not viewed as functions, but astransforms to be multiplied, the order that the transform names appearin stays the same: BtoC×BtoA→AtoC, e.g. matrix multiplication obeys theright to left rules of functional composition. While it can be arguedthat the other way around makes more logical sense, e.g. AtoB×BtoC→AtoC,in this document we will use the standard mathematical order inequations.

A note on inverse matrices. The traditional notation for the inverse ofa matrix (or a transform) is the name of the matrix to the minus onepower, e.g. if you have the matrix M, its inverse is M⁻¹. However, oneof the advantages of our long naming style is that most transforms arenamed in the form of the spaces that they transform from and to, e.g.,for spaces A and B, the transform is generally AtoB. This means that theinverse usually has a well-defined name, in this case it would be BtoA.So in many cases we will just use the direct name of the inverse, ratherthan applying the inverse operator to the original matrix.

III.B Hexagonimania

This section offers a (brief) introduction to the properties of hexagonsand tilings of hexagons. We will need the equations derived here tocompare data about cones and their hexagonal tiling from differentsources that use different definitions of the size of cones, and thespacing of hexagonal lattices. Almost always, by hexagon, we mean a sixsided polygon in which all the sides have exactly the same length. Theexceptions are the “hexagonally shaped” cones of the retina, in whichnot only do not all the sides share the same length, but sometimes have5, 7, 8 or 9 sides instead of 6! (Statistically, the average number ofsides of retinal cones appears to be about 6.2.)

Individual Hexagons

The Orientation of a Hexagon

To apply the terms “width” or “height” to a hexagon, you have to knowwhich of the two main possible orientations of the hexagon are beingreferred to.

Definition of term: hexagon on its side

Definition of term: hexagon on its end

As shown in FIG. 21, we define a hexagon on its side as a regularhexagon 2110 “resting” with one side aligned with horizontal axis. Asshown in FIG. 22, we define a hexagon on its end as the case in which aregular hexagon 2210 is balanced on one of its corners.

To avoid ambiguity, it is better when possible to use non-orientationspecific terms, so we introduce several.

The Diagonals of a Hexagon

Definition of term: LongDiagonal

Definition of term: Short Diagonal

We define the LongDiagonal of a hexagon as the longest line segment ofany orientation that can be drawn across the hexagon. FIG. 23 shows thelong diagonal element 2320 of a hexagon on its side element 2310. FIG.24 shows the long diagonal element 2420 of a hexagon on its end element2410. We define the ShortDiagonal of a hexagon as the shortest linesegment of any orientation that can be drawn across the hexagon. FIG. 25shows the short diagonal element 2520 of a hexagon on its side element2510. FIG. 26 shows the short diagonal element 2620 of a hexagon on itsend element 2610. The numerical relationship between the ShortDiagonaland the LongDiagonal of a hexagon is:

$\begin{matrix}{\frac{ShortDiagonal}{LongDiagonal} = \frac{\sqrt{3}}{2}} & (2) \\{{ShortDiagonal} = {\frac{\sqrt{3}}{2} \cdot {LongDiagonal}}} & (3) \\{{LongDiagonal} = {\frac{2}{\sqrt{3}} \cdot {ShortDiagonal}}} & (4)\end{matrix}$Tilings of HexagonsThe Orientation of Tilings of Hexagons

Definition of term: tiling of hexagon on their sides

Definition of term: tiling of hexagon on their ends

Just as lone hexagons have two main possible orientations, the same istrue when they gang up. In FIG. 27, we show a tiling of hexagons ontheir sides 2710, and in FIG. 28, we show a tiling of hexagons on theirends 2810.

The Pitch Between Hexagons in Tilings of Hexagons

Definition of term: long pitch

Definition of term: short pitch

Regardless of the orientation of a tiling of hexagons, there are twodifferent “pitches” to a hexagonal tiling. In FIG. 29 we define thelonger of the two: the long pitch 2920 of a hexagonal tiling 2910 is thedistance between left right neighboring hexagons in a tiling when thattiling is viewed as a tiling of hexagons on their end. In FIG. 30 wedefine the shorter of the two: the short pitch 3020 of a hexagonaltiling 3010 is the distance between adjacent hexagons diagonally in atiling when that tiling is viewed as a tiling of hexagons on their end.FIG. 30 shows diagonals up and to the right, and also the symmetricallyequivalent diagonals up and to the left, and those vertically apart,which all have the same short pitch. The long pitch turns out to be thesame as the short diagonal of a hexagon, so we will generally just usethe term short diagonal instead. The relationship between the ShortPitchand the ShortDiagonal is:

$\begin{matrix}{{ShortPitch} = {{\frac{3}{4} \cdot {LongDiagonal}} = {\frac{\sqrt{3}}{2} \cdot {ShortDiagonal}}}} & (5) \\{{ShortDiagonal} = {\frac{2}{\sqrt{3}} \cdot {ShortPitch}}} & (6)\end{matrix}$The Rows and Semi-Columns of Tilings of Hexagons on their End

Hexagonal tilings don't have the simple row-column relationship thattilings of squares (and rectangles) do. However, we can define someclose equivalents.

The Rows of Tilings of Hexagons on their End

In a tiling of hexagons on their end, we can see what looks somethinglike the rows of squares in a tiling of squares, but in the hexagonaltiling, ever other row is offset by half a hexagon from its neighborsabove and below. We will call these rows of hexagons (in a tiling ofhexagons on their end), but will differentiate between even and oddrows. FIG. 31 shows examples of the even rows of a tiling of hexagons ontheir end, element 3110. FIG. 32 shows examples of the odd rows of atiling of hexagons on their end, element 3210. We will call thebottommost even row of hexagons the first even row, as shown in FIG. 33element 3310. We will call the bottommost odd row of hexagons the firstodd row, as shown in FIG. 34 element 3410.

The Semi-Columns of Tilings of Hexagons on their End

One has to squint a bit harder, but a partial equivalent to columnsexist in tilings of hexagons on their end: semi-columns. FIG. 35 showsthe first semi-column 3510 of a tiling of hexagons on their end. FIG. 36shows the second semi-column 3610 of a tiling of hexagons on their end.

We can now assign semi-column numbers to both even and odd rows oftilings of hexagons on their end. FIG. 37 shows numbers (element 3710)for the first eight semi-columns for even rows of hexagons on their end.FIG. 38 shows numbers (element 3810) for the first eight semi-columnsfor odd rows of hexagons on their end.

Integral Row, Semi-Column Addresses of Tilings of Hexagons on their End

Numbering the first even row of a tiling of hexagons on their end 0,then the first odd row as 1, and so on, we can finally assign a unique2D integral address to each hexagon. This is shown in FIG. 39. Forexample, element 3910 is the hexagon with the unique 2D integral addressof (0,0).

Conversion from Rectangular to Hexagonal Coordinates

This sub-section will define how 2D (u, v) points from Cartesiancoordinate system on

2 can be identified with a particular unique (uu, vv) integral hexagonaddress. For this definition we will assume that the within the uvCartesian space each hexagonal pixel shape on its end has a width ofunity, e.g. the short diagonal is one.

Because the width of all hexagons is one, all the points within all theleftmost hexagons of all even rows have a u address in the range of [01). All points within all the second leftmost hexagons in all the evenrows have a u address in the range of [1 2), and so on. We associatewith each hexagon in even rows an integer uu semi-column address that isderived from the floor[ ] of their u address range. This is the first ofthe unique two integer pixel address for even row hexagons.

The situation with all odd row hexagons is similar, but shifted to theright by ½. Thus all the points within all the leftmost hexagons of alleven rows have a u address in the range of [½ 1½). All points within allthe second leftmost hexagons in all the odd rows have a u address in therange of [1½ 2½), and so on. We associate with each hexagon in odd rowsan integer uu semi-column address that is derived from the floor[ ] oftheir u address range. This is the first of the unique two integer pixeladdress for odd row hexagons. This is shown graphically in FIG. 40,where the height of a unit-wide hexagon on its end is shown as2/√{square root over (3)} (element 4010).

As shown in FIG. 41, the pitch 4110 between adjacent even and odd rowsof hexagons is the short pitch of the tiling, which for a unit shortdiagonal hexagon is √{square root over (3)}/2. Consider the set ofhorizontal lines with v axis intercept of n·√{square root over (3)}/2,where n is a positive integer. Just as the positive u axis contains thebottom most points of all the hexagons in the first row, the horizontallines parameterized by n contain the bottommost points of all thehexagons of the n-th row, starting with n=0 (the positive u axis), forboth even and odd rows of hexagons. These lines are also shown in FIG.41, as element 4120. The integer n is the integer vv address for eachhexagon, which is the second of the unique two integer pixel address forhexagons. This completes the assigning of unique two integer pixeladdresses to each hexagon in the tiling.

Each point (u v) in the Cartesian space lies within a unique hexagon ofthe tiling, which is pretty much the definition of “tiling the plane.”The algorithm for converting a given (u v) Cartesian space point to aunique (uu vv) hexagon id is straight forward, if slightly laborious todetail. The algorithm is well known from the literature, so it will notbe repeated here. The function CarToHex[ ] is the notation used toindicate this process. Thus we have:uu[u,v]=CarToHex[u,v]·uu  (7)vv[u,v]=CarToHex[u,v]·vv  (8)

The equivalent integer pixel address computing for a square tiling is(floor[u] floor[v]). Because the function is destructive, no (unique)inverse exists.

Properties of Hexagons and Tilings of Hexagons

The Area of a Hexagon

The area of a hexagon can be expressed either in terms of itsShortDiagonal, its LongDiagonal, or its ShortPitch.

$\begin{matrix}\begin{matrix}{{area} = {\frac{3}{4} \cdot {LongDiagonal} \cdot {ShortDiagonal}}} \\{= {{\frac{3}{8} \cdot \sqrt{3} \cdot {LongDiagonal}^{2}} = {\frac{\sqrt{3}}{2} \cdot {ShortDiagonal}^{2}}}}\end{matrix} & (9) \\{{area} = {{{ShortPitch} \cdot {ShortDiagonal}} = {\frac{2}{\sqrt{3}} \cdot {ShortPitch}^{2}}}} & (10)\end{matrix}$

Sometimes we need to know the diameter of a circle with the same area asa hexagon, or vice versa. Narrowly defining the diameter of a hexagon tobe the diameter of the circle with the same area as a hexagon, we canrelate these:

$\begin{matrix}{\mspace{79mu}{{area} = {\pi \cdot \left( \frac{diameter}{2} \right)^{2}}}} & (11) \\{\mspace{79mu}{{diameter} = \sqrt{\frac{4 \cdot {area}}{\pi}}}} & (12) \\{{diameter} = {\sqrt{\frac{4 \cdot \frac{\sqrt{3}}{2} \cdot {ShortDiagonal}^{2}}{\pi}} = {\sqrt{\frac{2 \cdot \sqrt{3}}{\pi}} \cdot {ShortDiagonal}}}} & (13) \\{\mspace{79mu}{{ShortDiagonal} = {\sqrt{\frac{\pi}{2 \cdot \sqrt{3}}} \cdot {diameter}}}} & (14) \\{\mspace{79mu}{{ShortPitch} = {{\sqrt{\frac{\pi\sqrt{3}}{8}} \cdot {diameter}} \approx {0.85 \cdot {diameter}}}}} & (15) \\{\mspace{79mu}{{diameter} = {\sqrt{\frac{8}{\pi \cdot \sqrt{3}}} \cdot {ShortPitch}}}} & (16)\end{matrix}$

Many times, rather than be given an area, we are given a density ofhexagons per unit area. We need to be able to convert density to andfrom our linear measurements:

$\begin{matrix}{{density} = \frac{1}{area}} & (17) \\{{density} = {\frac{\sqrt{3}}{2} \cdot \frac{1}{{ShortPitch}^{2}}}} & (18) \\{{density} = {\frac{2}{\sqrt{3}} \cdot \frac{1}{{ShortDiagonal}^{2}}}} & (19) \\{{density} = {{\frac{1}{\pi} \cdot \left( \frac{2}{diameter} \right)^{2}} = \frac{4}{\pi \cdot {diameter}^{2}}}} & (20)\end{matrix}$

Inverting:

$\begin{matrix}{{area} = \frac{1}{density}} & (21) \\{{ShortPitch} = \sqrt{\frac{\sqrt{3}}{2} \cdot \frac{1}{density}}} & (22) \\{{ShortDiagonal} = \sqrt{\frac{2}{\sqrt{3}} \cdot \frac{1}{density}}} & (23) \\{{diameter} = \sqrt{\frac{4}{\pi \cdot {density}}}} & (24)\end{matrix}$Resolution of Hexagons Vs. Squares

Theorem: If you define the resolution of a given pixel shape to belimited by the lowest resolution cross-section of the pixel (e.g., aline across the pixel at any angle: choose the longest diagonal of thepixel shape), then hexagonal pixels are 30% more efficient than squarepixels. That is, to achieve at least a given worse case resolutionwithin a given region, 30% less hexagonal shaped pixels are needed thanrectangular shaped pixels.

Proof: The area of a hexagon whose LongDiagonal is one is ⅜·√{squareroot over (3)}≈0.65. (The area of a unit circle is

$\left. {\frac{\pi}{4} \approx {0.79.}} \right)$A square with me same worse case resolution is a square with a diagonalof length one. The area of a square with a diagonal of length one is ½.Thus the amount of area covered by a hexagon whose LongDiagonal is onerelative to the amount of area covered by a square with a diagonal oflength one is ¾·√{square root over (3)}≈1.3.Hexagonal Shaped Tilings of Hexagons

So far we have talked about hexagonal tilings that extend infinitelyacross the plane. But we will also need to create tilings of hexagonallyshaped groups of hexagonal tilings.

Definition of term: n-group of hexagons

We define an n-group of hexagons to be, for positive integer's n, asingle hexagon on its end surrounded by n layers of hexagons on theirend. A 0-group of hexagons thus consists of a single hexagon on its end.We illustrate this concept for n from 1 to 4 by example: FIG. 42 shows a1-group of hexagons 4210. FIG. 43 shows a 2-group of hexagons 4310. FIG.44 shows a 3-group of hexagons 4410. And FIG. 45 shows a 4-group ofhexagons 4510. In general, an n-group of hexagons will have 2·n+1hexagons across its widest row. The number of hexagons inside an n-groupof hexagons is given by the following equation:NumberOfHexagons=1+3·n·(n+1)  (25)

A very important property of n-group of hexagons on their end is thatthey tile the plane like hexagons on their side. This is illustrated inFIG. 46, where three 2-group of hexagons, 4610, 4620, and 4630, areshown partially tiling the plane. This prosperity will be used later totile a variable resolution space with several hexagonally shapedprojectors, each have hexagonal shaped pixels composed as an n-group ofhexagons.

Brief Notes on Square Tilings

The Orientation of a Square

Squares are often viewed as having only one orientation: flat end down(a square on its side: FIG. 47, element 4710). But a diamond orientationis perfectly reasonable: squares (oriented) on its end (FIG. 48, element4810). (The square in repose.) But we will deal mostly with squares ontheir side. Here the width and height are well defined, and are ofcourse the same.

The Diagonals of a Square

Like many other pixel shapes, the square has by our definition both along diagonal and a short diagonal. The long diagonal is just thediagonal of the square; the short diagonal of a square is just one edge.FIG. 49 shows the long diagonal element 4920 of a square on its sideelement 4910. FIG. 50 shows the long diagonal element 5020 of a squareon its end element 5010. FIG. 51 shows the short diagonal element 5120of a square on its side element 5110. FIG. 52 shows the short diagonalelement 5220 of a square on its end element 5210. The ratio of the longdiagonal to the short diagonal is √{square root over (2)}.

The Pitches of a Tilling of Squares

The long pitch of a tilling of squares is the same as its shortdiagonal: just the length of a side (FIG. 53 element 5320). The shortpitch of a tilling of squares is half the length of the long diagonal:√{square root over (2)}/2≈0.707 (FIG. 54 element 5420). This also theratio between the short diagonal and the long diagonal.

The Area of a Square

The area of a square is the square of its side length.

III.C. Mathematical Definition of the Viewsphere and Related Concepts

INTRODUCTION

The following describes a combined mathematical coordinate system modelfrom both the visual sciences (“object space,” “visual space”) andcomputer graphics (“camera space”, “view space”). The model isinherently variable resolution, and can be used both to describe thevariable resolution cones and retinal receptor fields of the human eyeand organization of the visual cortex as well as variable resolutioncontact lens displays and variable resolution computer graphicsrenderers.

Several spaces and coordinate systems will be described, defined bytheir mapping to ViewSpace. To model variable resolution spaces, and howthey map to ViewSpace, we will introduce a general class of manifoldscalled ScreenSurfaces.

Formalized Mappings: Manifolds

ScreenSurfaces are Manifolds

For the general class of objects we are about to define asScreenSurfaces, we need to define more than a mapping function toViewSpace. In fact, we will need the full semantics of what are calledmanifolds. Thus ScreenSurfaces are defined to be manifolds.

Manifolds

A manifold is a mathematical object, modeled on an affine space X, withtwo associated parts. In our case, the manifold will always be called aScreenSurface, or referred to by a name assigned to mean a particularScreenSurface: a named ScreenSurface. The affine space X will always beViewSpaceVS, which will be defined to be a subset of ViewSpace. Thefirst part of a manifold is a Hausdorff topological space M, which inour case will be a sub-portion of

2 called the surface of the ScreenSurface manifold. The second part of amanifold is an atlas containing one or more charts, in our case, usuallyonly one or two charts. A chart is a pair consisting of an open set inthe ScreenSurface surface, and a map ScreenSurfaceToViewSpaceVS(shorthand name sv) that takes points from the open set to ViewSpaceVS.Manifolds also have a degree of continuity; here we shall only assumethat a first derivative exists, except at the boundaries between charts(this last is not standard). The first element of the atlas will be thegeneral variable resolution mapping to ViewSpaceVS. If a second chartexists, it will always be referred to as the EndCap chart. It will bedefined on its own open set in the ScreenSurface surface, and a mapEndCap·ScreenSurfaceToViewSpaceVS (shorthand EndCap·sv) that takespoints from the end-cap's open set to ViewSpaceVS. This second chartwill always be a polar end-cap to, among other things, cover thedegeneracy that occurs in spherical coordinates at the north pole.

The ScreenSurface's surface is the surface on which we wish to performpixel and sample rendering for computer graphics applications, or imageanalysis on for biological modeling and image processing applications.

Both mappings have defined inverses: ViewSpaceVSToScreenSurface(shorthand name vs) that takes points from ViewSpaceVS to their opensets.

While formally each part of the ScreenSurface should be referred to as<ScreenSurface name>.<part-name>, sometimes, in context, theScreenSurface name will be used to refer to the part.

So a named ScreenSurface will primarily be used as a (formal) namedmapping.

Named Objects of ViewSpace

This section will describe several spaces, surfaces, and mappingsrelated to ViewSpace, which is just a well-defined form of object space.These will be used both for visual science purposes for real eyeballs,and to describe the semantics of the generalized computer graphics viewmodel.

Definition of term: ViewSpace

In the model described here, the term ViewSpace takes the place of thetraditional “camera space” of computer graphics. It can also beconsidered a well-defined form of visual science object space. Thecoordinate axis of ViewSpace are the same as the traditional “cameraspace”: the X and Y axis represent horizontal and vertical directions,the positive Z axis represents the general direction of greater depthfrom the origin. Because positive values of Z represent distance,ViewSpace is a left handed coordinate system. The origin of ViewSpacewill be the EyePoint or ViewPoint of the perspective transform that willbe defined. In the following, the character c will be used as ashorthand for ViewSpace, because here ViewSpace is the replacement ofthe older concept of camera space. The individual coordinate componentsof ViewSpace will be denoted by x, y, and z. (Sometimes, when anotherspace is involved that also used named coordinates x, y, and sometimesz, individual coordinate components of ViewSpace will be denoted by X,Y, and Z for clarity.)

Definition of term: ViewSphere

The ViewSphere is defined to be the two dimensional closed surface ofthe unit sphere in ViewSpace centered at the origin. The ViewSphere isthe surface upon which the three dimensional world of objects inViewSpace coordinates usually will be projected onto. No specialcharacter is defined for this space, as it is an abstract space,character names will be assigned to particular coordinate systemsdefined on the surface of the ViewSphere. (Even though the ViewSphere asdefined is a surface, we still will usually refer to “the surface of theViewSphere” to reinforce this point.)

A three dimensional embedding of the ViewSphere into three dimensionalViewSpace will be defined; see ViewSpaceVS. One existing standardreference two dimensional coordinate system will be defined for thesurface of the ViewSphere; see VisualCoordinates.

Definition of term: VisualField

We define the term VisualField, in the context of any chart with anassociated mapping to the ViewSphere, to be region of the ViewSpheresurface that is the image (range) of the mapping (as limited by itschart). The term VisualField, in the context of a particular atlas ofcharts, refers to the region of the ViewSphere surface that is the unionof the images of mappings (constrained by their charts) for all chartsin the specific atlas. In a mapping modeling the human eye, theVisualField would be just the visual field of the eye, with the outeredge of the valid region on the ViewSphere being the ora serrata (astransformed into visual eccentricity from retinal eccentricity). Insimpler cases the outer edge will be defined by a constant angle ofvisual eccentricity, though the projection mapping of traditionalrectangular external displays are bounded by “barrel distorted” sort ofrectangle on the surface of the ViewSphere. The VisualField of any givenmapping onto the ViewSphere almost always will map to less than thetotal 47c steradian surface area of the entire ViewSphere. Thisremaining region of the ViewSphere still exists, it is just consideredoutside the particular VisualField.

Definition of term: ViewSpaceVS

The ViewSpaceVS is defined to be the set of three dimensional points inViewSpace that are the embedding of the two dimensional points of theViewSphere into three dimensional ViewSpace. (We could have called thisspace “ViewSphereEmbededInViewSpace,” but for a shorter name, we let the“VS” after “ViewSpace” stands for “ViewSphere.”) In the following, thecharacter v will be used as a shorthand for ViewSpaceVS. The individualcoordinate components of ViewSpaceVS will be denoted by x, y, and z.

Definition of term: ViewSpaceToViewSpaceVS

Definition of term: cv

ViewSpaceToViewSpaceVS is the mapping of points from anywhere inViewSpace to points in ViewSpaceVS. The mapping from three dimensionalpoints in ViewSpace to three dimensional points in ViewSpaceVS is manyto one. But if the mapping ViewSpaceToViewSpaceVS is used to map all ofViewSpaceVS, the results is just exactly ViewSpaceVS again, which is thedefinition of a projection mapping (even though it ends in a threedimensional space, not a two dimensional space), and in particular aperspective mapping. The mapping itself is straightforward: treat all 3Dpoints in ViewSpace as vectors from the origin; the normalization of anysuch vector produces the desired mapped corresponding point inViewSpaceVS. Note that ViewSpaceToViewSpaceVS is not a normalization, itis a mapping from ViewSpace to ViewSpace that happens to be definedusing normalization. Given that ViewSpaceToViewSpaceVS is rather awkwardto use in equations, we will let the shorthand cv stand for it. To avoidconfusion, in the equation below we will use XYZ for the location of theoriginal point in ViewSpace. Now cv is defined as:ViewSpaceToViewSpaceVS·x[X,Y,Z]=X/√{square root over (X ² +Y ² +Z²)}  (26)ViewSpaceToViewSpaceVS·y[X,Y,Z]=Y/√{square root over (X ² +Y ² +Z²)}  (27)ViewSpaceToViewSpaceVS·z[X,Y,Z]=Z/√{square root over (X ² +Y ² +Z²)}  (28)

Because of the destructive nature of the projective transform, we can'treverse this mapping. But we will note that points in ViewSpaceVS are bydefinition points in ViewSpace.

Note our convention here of denoting the individual components of amapping by {mapping-name}. {component-name}. In some future cases, two(or more) different sets of coordinate component names will be used fordifferent coordinate frames defined on the same space. Most of thesewill be defined on two dimensional coordinate frames. Alternatecoordinate component names will never be used for changes of coordinateframe that change the number of dimensions.

The following will define one specific and one general two dimensionalcoordinate systems on the ViewSphere. Transforms will be defined betweenthese, as well as transforms from either two dimensional space toViewSpaceVS and to ViewSpace.

Definition of term: VisualCoordinates

Definition of term: eccentricity

We inherit the VisualCoordinates system from the Visual Sciences. It isa two dimensional longitude ϕ and a co-latitude θ coordinate systemdefined on the ViewSphere. The co-latitude θ is called the eccentricity.Visual coordinates differ from the standard convention for sphericalcoordinates in two ways. First, rather than “latitude,” “co-latitude” isused. This means that θ=0° is not at the equator, but at the north pole.Second, the usual convention is that the variable θ is the longitude,and that ϕ is the latitude. Visual coordinates reverses these two,because the eye is mostly symmetrical in longitude, thus the main angleof interest is generally the co-latitude (eccentricity), and thevariable θ is used for that, while ϕ is used for longitude. In thefollowing, the character z will be used as a shorthand forVisualCoordinates. The two coordinate components for VisualCoordinateswill be ϕ and θ. Note that VisualCoordinates is just a coordinate framefor the points from the 2D space of the ViewSphere, not the coordinateframe.

Definition of term: ViewSpaceVSToVisualCoordinates

Definition of term: vz

Given the above definitions, following our naming conventions, themapping that takes 3D points from ViewSpaceVS to 2D points on theViewSphere as represented in the VisualCoordinates system is namedViewSpaceVSToVisualCoordinates. Its shorthand name is vz. The definitionis:vz·ϕ[x,y,z]=a tan 2[y,x]  (29)vz·θ[x,y,z]=cos⁻¹[z]  (30)

where a tan 2[y,x] is the is the angular component of the rectangularcoordinates to polar coordinates transform of x and y.

Convention:

while most implementations of a tan 2[y,x] return results in the rangeof [−π+π), in this document the a tan 2[ ] function is defined to returnresults in the range of [0 2π). This is because we will many timesspecifically need the results to be in this form, but the conversionfrom the range of [−π+π), to [0 2π) involves conditionals: if θ>=0, thenθ, else 2π+θ. For the times when this specific form is not needed,either form will work, so we will just always use the [0 2π) form.

Definition of term: VisualCoordinatesToViewSpaceVS

Definition of term: zv

The inverse mapping of ViewSpaceVSToVisualCoordinates is

VisualCoordinatesToViewSpaceVS, with a shorthand name of zv. Thedefinition of this mapping is:zv·x[ϕ,θ]=cos [ϕ]·sin [θ]  (31)zv·y[ϕ,θ]=sin [ϕ]·sin [θ]  (32)zv·z[ϕ,θ]=cos [θ]  (33)

Note again that the VisualCoordinates are not “the” ViewSpherecoordinates; points on the ViewSphere surface can be expressed in manydifferent 2D coordinate systems. The one-to-one mapping defined by theequations above show that both xyz ViewSpaceVS coordinates as well as ϕθVisualCoordinates can be used to represent points located on theViewSphere surface. (Why didn't we just define xyz as an alternatetransformation of VisualCoordinates, and not have to have createdViewSpaceVS? Because this breaks the restriction on coordinate namescausing a change in the dimensionality of the space. This is whyVisualCoordinates exists as a separate space from any 2D space definedon the surface of the ViewSphere.)

Definition of term: ScreenSurface

As described above, a ScreenSurface is a manifold whose primary use isto define how points are mapped from the ScreenSurface's two dimensionalsurface to ViewSpaceVS (or a surrogate for ViewSpaceVS, e.g.VisualCoordinates), as well as the inverse mapping. The chart aspects ofthe manifold only come up when dealing with the EndCap portion of themanifold.

The two dimensional surface associated with a ScreenSurface manifold iscalled a surface rather than a plane because more complex screengeometries will be considered than the simple planer model of thetraditional (simple) computer graphics view model. Another way of sayingthis is that there will not always exist a linear, let alone planer,embedding of a ScreenSurface's surface into ViewSpace. In the following,the character s will be used as a shorthand for the genericScreenSurface. The two coordinate components for the ScreenSurfacessurface will usually be the pair u and v, denoting that the (0, 0)origin of the coordinate system is located at the lower left of thespace, but the pair x and y will be used to denote a transformedcoordinate system in which the (0, 0) origin is located at the center ofthe space, and the pair radius and angle will be used to indicate atransformed general polar coordinate system (with a centered origin).(See ScreenSurfaceCoordinates.) We will never use θ to denote the angleof a polar coordinate system: θ will always denote eccentricity (thoughnote that visual eccentricity and retinal eccentricity are not exactlythe same).

Definition of term: ViewSpaceVSToScreenSurface

Definition of term: vs

The mapping that takes points from ViewSpaceVS to points on theScreenSurface. Its shorthand name is vs. The mapping will have differentdefinitions depending on details of the form of the perspectiveprojection being employed, expressly including possibilities forprojections points on the ScreenSurface will have considerable variableresolution on the ViewSphere.

Definition of term: ScreenSurfaceToViewSpaceVS

Definition of term: sv

The inverse mapping of ViewSpaceVSToScreenSurface is

ScreenSurfaceToViewSpaceVS, with a shorthand name of sv.

Definition of term: VisualCoordinatesToScreenSurface

Definition of term: zs

The mapping that takes points from VisualCoordinates to points on theScreenSurface is VisualCoordinatesToScreenSurface. Its shorthand name iszs. This is the portion of the mapping ViewSpaceVSToScreenSurface thatstarts from VisualCoordinates rather than from ViewSpaceVS. Therelationship is:ViewSpaceVSToScreenSurface=VisualCoordinatesToScreenSurface×ViewSpaceVSToVisualCoordinatesThus:VisualCoordinatesToScreenSurface=ViewSpaceVSToScreenSurface×VisualCoordinatesToViewSpaceVSzs=vs×zv

Definition of term: ScreenSurfaceToVisualCoordinates

Definition of term: sz

The inverse mapping of VisualCoordinatesToScreenSurface isScreenSurfaceToVisualCoordinates, with a shorthand name of sz. This isthe portion of the mapping ScreenSurfaceToViewSpaceVS that stops atVisualCoordinates, rather than going all the way to ViewSpaceVS. Therelationship is:ScreenSurfaceToViewSpaceVS=VisualCoordinatesToViewSpaceVS×ScreenSurfaceToVisualCoordinatesThus:ScreenSurfaceToVisualCoordinates=ViewSpaceVSToVisualCoordinates×ScreenSurfaceToViewSpaceVSsz=vz×sv

By using the ViewSpaceToViewSpaceVS transform, we can define transformsfrom ViewSpace to either of the 2D ViewSphere coordinate systems.Because no inverse of ViewSpaceToViewSpaceVS exists, these newtransforms also do not have inverses.

Definition of term: ViewSpaceToVisualCoordinates

Definition of term: cz

The mapping that takes points from ViewSpace to points on theVisualCoordinates.

The shorthand name for ViewSpaceToVisualCoordinates is cz. It is definedas the multiplication of ViewSpaceVSToVisualCoordinates withViewSpaceToViewSpaceVS, and is given by:

$\begin{matrix}{\phi = {{{cz} \cdot {\phi\left\lbrack {x,y,z} \right\rbrack}} = {{atan}\;{2\left\lbrack {y,x} \right\rbrack}}}} & (34) \\{\theta = {{{cz} \cdot {\theta\left\lbrack {x,y,z} \right\rbrack}} = {\cos^{- 1}\left\lbrack \frac{z}{\sqrt{x^{2} + y^{2} + z^{2}}} \right\rbrack}}} & (35)\end{matrix}$

No (unique) inverse exists.

Definition of term: ViewSpaceToScreenSurface

Definition of term: cs

The mapping that takes points from ViewSpace to points inVisualCoordinates. The shorthand name for ViewSpaceToScreenSurface iscs. It is defined as the multiplication of ViewSpaceVSToScreenSurfacewith ViewSpaceToViewSpaceVS. No inverse exists.

Summary of Mapping Notation

Single characters have been associated with the four spaces associatedwith the view mapping (along with hints to remember them):

c: ViewSpace (‘c’ because ViewSpace is similar to “camera space”) v:ViewSpaceVS (‘v’ because this is the “view” space we most use) z:VisualCoordinates (‘z’ because “visual” is pronounced “vizual”) s:ScreenSurface (‘s’ for “screen”)

Now the nine transformations between the spaces are just the appropriatepairing of the individual characters for the spaces:

cv=ViewSpaceToViewSpaceVS

cz=ViewSpaceToVisualCoordinates

cs=ViewSpaceToScreenSurface

vz=ViewSpaceVSToVisualCoordinates

zv=VisualCoordinatesToViewSpaceVS

vs=ViewSpaceVSToScreenSurface

sv=ScreenSurfaceToViewSpaceVS

zs=VisualCoordinatesToScreenSurface

sz=ScreenSurfaceToVisualCoordinates

Screen Surface Coordinates

Definition of term: ScreenSurfaceCoordinates

Definition of term: ScreenWidth

Definition of term: SW

Definition of term: ScreenHeight

Definition of term: SH

The topological space associated with every ScreenSurface manifold isalways a portion of

2. Most of the time, that portion is a fixed size rectangle. Here, whenthis is true, we define four specific 2D ScreenSurfaceCoordinatessystems on that rectangular surface portion. What is important about twoof these four coordinate systems is that they define integralspecifically shaped equal size pixels to their tiling of theScreenSurface's surface: they define pixel spaces.

2D Cartesian uv Lower Left Origin Coordinate System

The first is a two dimensional Cartesian uv coordinate. The range of theu coordinate is defined to be [0, ScreenWidth), where ScreenWidth willcommonly be abbreviated to SW. The range of the v coordinate is definedto be [0, ScreenHeight), where ScreenHeight will commonly be abbreviatedto SH. Thus this is a rectangular portion, but in many timestopologically it will be defined to be a strip: for any given vcoordinate, for a given u coordinate u₀ in the range [0, ScreenWidth),all u coordinates outside of the range [0, ScreenWidth) for which u₀=umod ScreenWidth are the same points. Note that in this coordinatesystem, the origin (0,0) is located at the lower left corner of therectangle.

2D Cartesian xy Centered Origin Coordinate System

The second is an offset version of the first coordinate system. Todifferentiate it from the first, the coordinate components are x and y.The range of the x coordinate is defined to be [−ScreenWidth/2,ScreenWidth/2). The range of the y coordinate is defined to be[−ScreenHeight/2, ScreenHeight/2). This coordinate system is still arectangular portion, but the origin (0, 0) is located at the center ofthe rectangle. It is related to the previous coordinate system via thetransforms:x[u,v]=u−SW/2  (36)y[u,v]=v−SH/2  (37)and the inverse is:u[x,y]=x+SW/2  (38)v[x,y]=y+SH/2  (39)

The coordinate system is an intermediate one not used for purposes ofrendering, so integral pixels are not defined for it. (This would haveinvolved negative integer pixel coordinates, which we wish to avoid.)

2D Polar Angle Radius Coordinate System

The third coordinate system is the polar form of the second coordinatesystem, with coordinate components angle and radius. The range of theangle coordinate is defined to be [−π, +π) (and wraps around past this).The range of the radius coordinate is defined to be [0, ScreenHeight).The origin of this coordinate system is the same as that of the xycoordinate system. However, this coordinate system is a circular portionof the ScreenSurface. It is related to the xy coordinate system via thetransforms:angle[x,y]=a tan 2[y,x]  (40)radius[x,y]=√{square root over (x ² +y ²)}  (41)and the inverse is:x[angle,radius]=radius·cos [angle]  (42)y[angle,radius]=radius·sin [angle]  (43)

The coordinate system is an intermediate one not used for purposes ofrendering, so integral pixels are not defined for it. (This would haveinvolved quantizing the angle, which would be contrary to the purposesfor which the coordinate system was created.)

2D Hexagonal Lower Left Origin Coordinate System

The fourth specific 2D ScreenSurfaceCoordinates coordinate systemdefined on a rectangular portion of the surface is a hexagonal grid. Itis a tiling of hexagons on their end, relative to the Cartesian uv spaceorientation. All pixels have the same shape and size (within thecoordinate system). In this case, all the pixels have a hexagonal shape.Setting the scale of the hexagonal pixels relative to the uv Cartesianspace is a complex issue, and will be left to be described later.

The mapping from Cartesian u v space coordinates to uu vv hexagonal idsis as described by equations (7) and (8).

There are some “holes” in this tiling on the four sides of the tiling ofthe rectangle. If, as is many times the case, the u address “wrapsaround” at SW, then the holes on the left and right side are covered.The “holes” on the top and bottom of the rectangle are usually dealtwith by the domain side of the chart with the associated mapping that isusing hexagonal coordinates.

Again, the hexagons described in the address labeling method werearbitrarily chosen to have unit width in the space we were working in.The precise scale change between the Cartesian uv space and the spacethat the hexagons are defined is not yet fixed.

Notation Notes

In the future, we will use our convention of letting the choice ofcoordinate component name determine which of these three coordinatesystems are being referred to. In some equations, that means that therewill be an implicit change of coordinate systems. We will always use thecoordinate name angle, and never use the symbol θ. Here θ is exclusivelyreserved to denote eccentricity.

In the future, we will be comparing different ScreenSurface mappings. Insuch cases, references to SW and SH will be ambiguous; we will use theterminology <ScreenSurface Name>·SH (and ·SW, ·u, . . . ), where<ScreenSurface Name> is a particular named ScreenSurface, to make itclear which parameters are being referred to.

III.D. Scale Changes Between the ScreenSurface and ViewSpaceVS

Note that three of our four coordinate systems (ViewSpace, ViewSpaceVS,VisualCoordinates) have a fixed definition, and thus the transformationsto and from any of the three to any other of the three are also fixed,e.g. we know what all these transforms are (when they exist). So far wehave not yet defined any specific (named) instance of a ScreenSurface:thus far it is only an abstraction. The whole point was to set upframework to examining properties (including visual properties) of anyarbitrary mapping (and related information) we are given to define aparticular ScreenSurface manifold.

Soon, we will define the all-important property of resolution in termsof distance on the surface of the ViewSphere. But first we will need todevelop a way to relate the size of a pixel in a particularScreenSurface surface (

2) to distance on the ViewSphere. Let d be a tangent vector in thetangent space to the ScreenSurface surface. If we were dealing with amapping ƒ from ScreenSurface to

1, the directional derivative ∇_(d)ƒ would tell us how much the changein scale would be between the derivative in

1 off in the direction d relative to unity (the normalized length of d).But unfortunately we want to compute this property for mappings,especially including sv, which maps to

3, not

1. So we will develop a differential operator for that scale change inhigher dimensional image (range). We will call this function thedirectional magnitude derivative, and denote it by ∇_(d)ƒ. This can bedistinguished from the ordinary directional derivative when the mapping(whether in bold or not) is known to be to

2 or

3, not

1. (When ƒ is a mapping to vectors, e.g. a vector field, the samenotation has been used to denotes a connection, but here ƒ is always amapping to points.) The closest existing mathematical concept to this isthe determinant of the Jacobian, but this measures the ratio of the areaor volume change caused by a mapping, not the absolute value of thelinear scale change in a given direction. So in the next subsection wewill go into a considerable amount of mathematical detail to develop thefunction we want. In the end, it will turn out to be expressible as thelength of a vector of ordinary directional derivatives. Once developed,we will mainly use the function symbolically, and only occasionallyactually compute it.

Tangent Spaces and how to Use them

(The following is summary of some of the material and mostly uses thenotation found in [Dodson, C. T. J., and Poston, T. 1997, TensorGeometry, The Geometric Viewpoint and its Uses., 2nd Ed., Springer].)

Given a space X, we will denote by d[x,y] the difference functionbetween two points x∈X and y∈X, and the image of d is a vector space,which we will denote by T. We will denote by d_(x)[y] the differencefunction between an arbitrary point y and always the point x, thisproduces a vector in the vector space T.

Given a space X and a point x∈X, the tangent space to X at x is denotedby T_(x)X. The contents of T_(x)X are the vectors d_(x)[y] for allpossible y∈X, thus T_(x)X is a space with the same number of dimensionsas X. The elements of T_(x)X are called tangent vectors.

Given two spaces, X and X′, which can have different numbers ofdimensions, and which have difference functions d and d′, let ƒ be amapping from X to X′. We denote a derivative off at x∈X, as D_(x)ƒ.D_(x)ƒ is a map from the tangent space T_(x)X to the tangent spaceT_(ƒ[x])X′. d′_(ƒ[x]) is the difference function between an arbitrarypoint in X′ and always the point ƒ[x]εX′, and produces a tangent vectorresult in the tangent vector space T′. We denote its inverse as d′⁻¹_(ƒ[x]). It is a map that takes a vector from T′ to a tangent vector inT_(ƒ[x])X′. Given t as a tangent vector in T_(x)X, we can now defineD_(x)ƒ[t] as:

$\begin{matrix}{{D_{x}{f\lbrack t\rbrack}} = {\lim\limits_{h\rightarrow 0}\mspace{14mu}{d_{f{\lbrack x\rbrack}}^{\prime - 1}\left\lbrack \frac{d^{\prime}\left\lbrack {{f\lbrack x\rbrack},{f\left\lbrack {x + {h \cdot t}} \right\rbrack}} \right\rbrack}{h} \right\rbrack}}} & (44)\end{matrix}$

Now we will impose the coordinate frames (b₀ . . . b_(n)) on X and (c₀ .. . c_(n)?) on X's. This means:

$\begin{matrix}{{D_{x}{f\left\lbrack b_{j} \right\rbrack}} = {\frac{\partial f^{i}}{\partial x^{j}} \cdot c_{i}}} & (45) \\{{D_{x}{f\lbrack t\rbrack}} = {\frac{\partial f^{i}}{\partial x^{j}} \cdot c_{i} \cdot t^{j}}} & (46)\end{matrix}$

Where ƒ^(i) are the coordinate component functions off, and t^(j) arethe coordinate components of the tangent vector t (and we are assumingthat the summation convention is being used.) The partial derivativesexpressed as a matrix is the Jacobian matrix of the map ƒ at x. We willonly be interested in maps from

2 to

2, from

2 to

2, and from

2 to

3, so we will write out all three:

$\begin{matrix}{{D_{x}{f\lbrack t\rbrack}} = \left. {\begin{bmatrix}{\frac{\partial f^{1}}{\partial x^{1}}\lbrack x\rbrack} & {\frac{\partial f^{1}}{\partial x^{2}}\lbrack x\rbrack} \\{\frac{\partial f^{2}}{\partial x^{1}}\lbrack x\rbrack} & {\frac{\partial f^{2}}{\partial x^{2}}\lbrack x\rbrack}\end{bmatrix}\mspace{14mu}{when}\mspace{14mu} f\text{:}2}\rightarrow{2} \right.} & (47) \\{{{D_{x}{f\lbrack t\rbrack}} = \begin{bmatrix}{\frac{\partial f^{1}}{\partial x^{1}}\lbrack x\rbrack} & {\frac{\partial f^{1}}{\partial x^{2}}\lbrack x\rbrack} & {\frac{\partial f^{1}}{\partial x^{3}}\lbrack x\rbrack} \\{\frac{\partial f^{2}}{\partial x^{1}}\lbrack x\rbrack} & {\frac{\partial f^{2}}{\partial x^{2}}\lbrack x\rbrack} & {\frac{\partial f^{2}}{\partial x^{3}}\lbrack x\rbrack}\end{bmatrix}}\mspace{14mu}\left. {{when}\mspace{14mu} f\text{:}3}\rightarrow{2} \right.} & (48) \\{{D_{x}{f\lbrack t\rbrack}} = \left. {\begin{bmatrix}{\frac{\partial f^{1}}{\partial x^{1}}\lbrack x\rbrack} & {\frac{\partial f^{1}}{\partial x^{2}}\lbrack x\rbrack} \\{\frac{\partial f^{2}}{\partial x^{1}}\lbrack x\rbrack} & {\frac{\partial f^{2}}{\partial x^{2}}\lbrack x\rbrack} \\{\frac{\partial f^{3}}{\partial x^{1}}\lbrack x\rbrack} & {\frac{\partial f^{3}}{\partial x^{2}}\lbrack x\rbrack}\end{bmatrix}\mspace{14mu}{when}\mspace{14mu} f\text{:}2}\rightarrow{3} \right.} & (49)\end{matrix}$

We now have the first part of what we want: given that we have a mappingƒ between two spaces, we have a function D_(x)ƒ[t] that, for a givenpoint x in the first space, will take us from a tangent vector to thefirst space t to a tangent vector to the second space. Now all that isleft is to take the ratio of the length of the second tangent vector tothe length of the first. First we note that the squared length of thesecond vector is:

$\begin{matrix}{{D_{x}{{f\lbrack t\rbrack} \cdot D_{x}}{f\lbrack t\rbrack}} = {{\left( {{\frac{\partial f^{i}}{\partial x^{j}}\lbrack x\rbrack} \cdot c_{i} \cdot t^{j}} \right) \cdot \left( {{\frac{\partial f^{k}}{\partial x^{l}}\lbrack x\rbrack} \cdot c_{k} \cdot t^{l}} \right)} = {\sum\limits_{i}\left( {\sum\limits_{j}{{\frac{\partial f^{i}}{\partial x^{j}}\lbrack x\rbrack} \cdot t^{j}}} \right)^{2}}}} & (50)\end{matrix}$

Definition of term: directional magnitude derivative

Definition of term: ∇_(d)ƒ

$\begin{matrix}{{{{\nabla_{d}{f\lbrack x\rbrack}} =}\frac{{D_{x}{f\lbrack x\rbrack}}}{d}} = {\frac{\sqrt{D_{x}{{f\lbrack d\rbrack} \cdot D_{x}}{f\lbrack d\rbrack}}}{\sqrt{d \cdot d}} = {\frac{1}{d} \cdot \sqrt{\sum\limits_{i}\left( {\sum\limits_{j}{{\frac{\partial f^{i}}{\partial x^{j}}\lbrack x\rbrack} \cdot t^{j}}} \right)^{2}}}}} & (51)\end{matrix}$

Equation (51) above is our formal definition of the directionalmagnitude derivative in the most general case. We have changed the nameof the tangent vector t to d, because it is now used as a tangentdirection vector (e.g., its magnitude doesn't matter anymore). (Notethat d is different from d, the difference function.) In the future, wewill elide the argument point x, because it is generally clear fromcontext.

One should note that where d′=D_(x)ƒ[d].

$\begin{matrix}{{\nabla_{d^{\prime}}f^{- 1}} = \frac{1}{\nabla_{d}f}} & (52)\end{matrix}$

This just means that when a tangent vector to the second space is mappedby ∇_(d′)ƒ⁻¹ back into a tangent vector to the first space at a givenpoint, the relative scale change is reversed.

It is interesting to note that we can alternately define the directionalmagnitude derivative ∇_(d)ƒ in terms of the ordinary directionalderivative. Here's the definition when the image off produces twodimensional points, and three dimensional points:∇_(d)ƒ=|

_(d)ƒ¹∇_(d)ƒ²

|  (53)∇_(d)ƒ=|

∇_(d) ¹∇_(d)ƒ²∇_(d)ƒ³

|  (54)

Where ∇_(d)ƒ^(k) is just the ordinary directional derivative on thescalar valued function ƒ^(k) in the direction d. Seeing that bothdefinitions involve a square root, it is clear that our directionalmagnitude derivative only produces positive results.

The ordinary directional derivative is defined in the tangent space tothe manifold described by ƒ. The vector ∇ƒ also lies on the tangentspace, and points in the direction of the steepest increase in ƒ at agiven point, with |∇ƒ| indicating the rate of the change in thisdirection. The ordinary directional derivative can be defined as the dotproduct of the vector ∇ƒ with the unit vector in the direction d, or|∇ƒ| times the cosine of an angle between ∇ƒ and d. At angles of 90°,e.g. when d is orthogonal to ∇ƒ, ∇_(d)ƒ will be 0; at angles greaterthan 90°, it will be negative.

In terms of differentials, the directional magnitude derivative can bethought of as the ratio between the length of a (infinitesimal step) inthe image off to the length of the (infinitesimal) step in the domainoff in the direction d that caused it. Not only is this always positive,but it can be greater than zero everywhere (or almost everywhere).

We will use a special case of the Jacobian. When we map the pixel unitvectors u and v in the tangent space to the ScreenSurface surface at athe ScreenSurface surface point p to their corresponding vectors u′ andv′ in the tangent space to ViewSpaceVS at the ViewSpaceVS point p′, then1/|u′×v′| is the density of ScreenSurface surface pixels at the point p′(assuming a square pixel tiling, the equation has to be adjusted forhexagonal tilings).

As was mentioned, we will frequently be using the directional magnitudederivative of the mapping sv, or its inverse vs. But the ViewSpaceVSdirection vector stated will sometimes be θ, e.g. ∇_(θ) vs. What doesthis mean? Well, it just means that we want use the direction of the θaxis, but expressed in ViewSpaceVS coordinates. That's just D_(x)zv[θ],and in ∇_(ϕ) vs ϕ will be D_(x)zv[ϕ]. Yes, in general these will not beunit length vectors in ViewSpaceVS, but since the vectors are only beingused for their directions, this isn't an issue.

Definition of term: visual longitudinal direction

Definition of term: longitudinal direction

Definition of term: ϕ

Definition of term: retinal longitudinal direction

We will use the phrase visual longitudinal direction, or justlongitudinal direction, or the boldface symbol ϕ, to indicate for agiven point on the ViewSphere, in what direction does the (local)longitude increase, and the eccentricity (locally) not change at all.When talking about the physical RetinalSphere, we will refer to theretinal longitudinal direction.

Definition of term: visual eccentricity direction

Definition of term: eccentricity direction

Definition of term: θ

Definition of term: retinal eccentricity direction

We will use the phrase visual eccentricity direction, or justeccentricity direction, or the boldface symbol θ, to indicate for agiven point on the ViewSphere, in what direction does the (local)eccentricity increase, and the longitude (locally) not change at all.When talking about the physical RetinalSphere, we will refer to theretinal eccentricity direction.

More Comments on the Semantics of the ViewSphere

In most physical eyes, the physical position of the retina is bestunderstood as a spherical imaging surface that corresponds to a portionof the rear portions of ViewSpaceVS; this is why there is an inversionof the image on the retina. However, consistent with the traditionalViewPlane, the mathematical convention is that the spherical viewsurface is in the front portions of ViewSpaceVS. What front portions ofViewSpaceVS is used is dependent on the “eye” being modeled. However,the limitation to points in the frontal portions is not a limitation topoints with positive z values, as the field of view of even a single eyecan include points of easily greater than 90° of eccentricity from thedirection of highest resolution, without implying that the field of viewis anywhere greater than, or even near 180°.

An Example ScreenSurface: The ViewPlane

Definition of term: ViewPlane

Definition of term: FOV

The “view plane” traditionally used in computer graphics will be ourfirst example of a ScreenSurface manifold. The mapping portion is calledplaner because it can be linearly embedded in ViewSpace as a section ofa plane. We will refer to this particular ScreenSurface as theViewPlane. The ViewSpace embedding of the ViewPlane is rectangularportion of a plane orthogonal to the z axis, and in simple view models,also centered on the z axis. In more complex view models, the center ofthe rectangle is offset from the z axis, see [Deering, M. 1992: HighResolution Virtual Reality. In proceedings of SIGGRAPH 1992, pp. 195,202] for a view model in which the decentering is based on head-trackingand stereo display. For the purposes of our example here, we will assumeno such z axis offset.

The horizontal angular field of view from the center of ViewSpacethrough the left and right edges of the rectangle is denoted by the termFOV (in units of radians). We will assume square pixels, e.g. that theaspect ratio of the rectangle defined in ScreenSurface coordinates SW/SHis the same as the aspect ratio defined in ViewSpace by sin [horizontalangular field of view/2]/sin [vertical field of view/2]. Thus thevertical field of view does not need to be specified, it is fixed as2·sin⁻¹[sin [FOV/2]·SH/SW].

The cs mapping for the ViewPlane is given by:

$\begin{matrix}{u = {{{ViewPlane}.{cs}.{u\left\lbrack {X,Y,Z} \right\rbrack}} = {{\frac{X}{Z} \cdot \frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack}} + \frac{SW}{2}}}} & (55) \\{v = {{{ViewPlane}.{cs}.{v\left\lbrack {X,Y,Z} \right\rbrack}} = {{\frac{Y}{Z} \cdot \frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack}} + \frac{SH}{2}}}} & (56)\end{matrix}$

The use of SW in the equation for cs·v is not a typo, it is aconsequence of mandating square pixels.

Occasionally, instead of uv, we will need to use an xy centeredcoordinate system. This shouldn't cause confusion, as the cs mapping bydefinition lands on the ScreenSurface, and we are just using atransformed version of the uv coordinates. In this case, the cs mappingis defined as (here for clarity we will use XYZ for the three ViewSpacecoordinates):

$\begin{matrix}{x = {{{ViewPlane}.{cs}.{x\left\lbrack {X,Y,Z} \right\rbrack}} = {\frac{X}{Z} \cdot \frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack}}}} & (57) \\{y = {{{ViewPlane}.{cs}.{y\left\lbrack {X,Y,Z} \right\rbrack}} = {\frac{Y}{Z} \cdot \frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack}}}} & (58)\end{matrix}$

We will also sometimes need to use the radius and angle polar coordinateframe for the ViewPlane ScreenSurface. Now the cs mapping is defined as:

$\begin{matrix}\begin{matrix}{{radius} = {{ViewPlane}.{cs}.{{radius}\left\lbrack {X,Y,Z} \right\rbrack}}} \\{= {\sqrt{{{cs}.{x\left\lbrack {X,Y,Z} \right\rbrack}^{2}} + {{cs}.{y\left\lbrack {X,Y,Z} \right\rbrack}^{2}}} = {\frac{{SW}/2}{{\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot Z} \cdot \sqrt{X^{2} + Y^{2}}}}}\end{matrix} & (59) \\\begin{matrix}{{angle} = {{ViewPlane}.{cs}.{{angle}\left\lbrack {X,Y,Z} \right\rbrack}}} \\{= {{a\;\tan\;{2\left\lbrack {{{cs}.{y\left\lbrack {X,Y,Z} \right\rbrack}},{{cs}.{x\left\lbrack {X.Y.Z} \right\rbrack}}} \right\rbrack}} = {a\;\tan\;{2\left\lbrack {Y,X} \right\rbrack}}}}\end{matrix} & (60)\end{matrix}$

Note that the angle defined here is the same longitude as defined in

VisualCoordinates:ViewPlane·cs·angle[X,Y,Z]=cz·ϕ[X,Y,Z]  (61)

As with the cv mapping, because of the destructive nature of the csmapping, no unique inverse for cs (sc) exists (though we will define aparticular one later). Note also that this mapping is defined only forfields of view less than 180°. As will be seen later, not all csmappings have this field of view limitation.

Because ViewSpaceVS is a subset of ViewSpace, the mapping for vs is thesame as that for cs:

$\begin{matrix}\begin{matrix}{u = {{{ViewPlane}.{vs}.{u\left\lbrack {X,Y,Z} \right\rbrack}} = {{ViewPlane}.{cs}.{u\left\lbrack {X,Y,Z} \right\rbrack}}}} \\{= {{\frac{X}{Z} \cdot \frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack}} + \frac{SW}{2}}}\end{matrix} & (62) \\\begin{matrix}{v = {{{ViewPlane}.{zs}.{v\left\lbrack {X,Y,Z} \right\rbrack}} = {{ViewPlane}.{cs}.{v\left\lbrack {X,Y,Z} \right\rbrack}}}} \\{= {{\frac{Y}{Z} \cdot \frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack}} + \frac{SH}{2}}}\end{matrix} & (63)\end{matrix}$

And to xy coordinates:

$\begin{matrix}{x = {{{ViewPlane}.{vs}.{x\left\lbrack {X,Y,Z} \right\rbrack}} = {{{ViewPlane}.{cs}.{x\left\lbrack {X,Y,Z} \right\rbrack}} = {\frac{X}{Z} \cdot \frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack}}}}} & (64) \\{y = {{{ViewPlane}.{vs}.{y\left\lbrack {X,Y,Z} \right\rbrack}} = {{{ViewPlane}.{cs}.{y\left\lbrack {X,Y,Z} \right\rbrack}} = {\frac{Y}{Z} \cdot \frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack}}}}} & (65)\end{matrix}$

And to radius angle coordinates:

$\begin{matrix}\begin{matrix}{{{ViewPlane}.{vs}.{{radius}\left\lbrack {X,Y,Z} \right\rbrack}} = {{ViewPlane}.{cs}.{{radius}\left\lbrack {X,Y,Z} \right\rbrack}}} \\{= \sqrt{{{cs}.{x\left\lbrack {X,Y,Z} \right\rbrack}^{2}} + {{cs}.{y\left\lbrack {X,Y,Z} \right\rbrack}^{2}}}} \\{= {\frac{{SW}/2}{{\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot Z} \cdot \sqrt{X^{2} + Y^{2}}}}\end{matrix} & (66) \\\begin{matrix}{{{ViewPlane}.{vs}.{{angle}\left\lbrack {X,Y,Z} \right\rbrack}} = {{ViewPlane}.{cs}.{{angle}\left\lbrack {X,Y,Z} \right\rbrack}}} \\{= {a\;\tan\;{2\left\lbrack {{{cs}.{y\left\lbrack {X,Y,Z} \right\rbrack}},{{cs}.{x\left\lbrack {X.Y.Z} \right\rbrack}}} \right\rbrack}}} \\{= {a\;\tan\;{2\left\lbrack {Y,X} \right\rbrack}}}\end{matrix} & (67)\end{matrix}$

However this time the inverse of the mapping, the mapping sv, doesexist. Defining the point P in ViewSpace coordinates using the uvViewPlane coordinates as:

$\begin{matrix}{P = \left\langle {{\frac{u - {{SW}/2}}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}},{\frac{v - {{SH}/2}}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}},1} \right\rangle} & (68)\end{matrix}$

or, alternately, using the centered xy ViewPlane coordinates:

$\begin{matrix}{P = \left\langle {{\frac{x}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}},{\frac{y}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}},1} \right\rangle} & (69)\end{matrix}$

or, alternately, using the polar radius angle ViewPlane coordinates:

$\begin{matrix}{P = \left\langle {{\frac{{radius} \cdot {\cos\lbrack{angle}\rbrack}}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}},{\frac{{radius} \cdot {\sin\lbrack{angle}\rbrack}}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}},1} \right\rangle} & (70)\end{matrix}$

and then using ViewSpaceToViewSpaceVS (cv), then sv is:

$\begin{matrix}{{x = {{{sv}.{x\left\lbrack {u,v} \right\rbrack}} = \frac{P.x}{P}}},{y = {{{sv}.{y\left\lbrack {u,v} \right\rbrack}} = \frac{P.y}{P}}},{z = {{{sv}.{z\left\lbrack {u,v} \right\rbrack}} = \frac{P.z}{P}}}} & (71)\end{matrix}$

The ViewPlane can be embedded into ViewSpace as a plane orthogonal tothe z axis, but at any arbitrary point on the z axis. The z value ofthis point determines the scale of the mapping. In the past, thecharacter D (roughly analogous to focal length) was sometimes used todenote this z distance. Using D as a parameter, a family of mappingsfrom ViewPlane (a ScreenSurface) to ViewSpace, sc, exists as anembeddings:

$\begin{matrix}{x = {{{sc}.{x\left\lbrack {u,v} \right\rbrack}} = {{\frac{u - {{SW}/w}}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot D} = {\frac{x}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot D}}}} & (72) \\{y = {{{sc}.{x\left\lbrack {u,v} \right\rbrack}} = {{\frac{v - {{SW}/2}}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot D} = {\frac{y}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot D}}}} & (73) \\{x = {{{sc}.{x\left\lbrack {u,v} \right\rbrack}} = D}} & (74)\end{matrix}$

(Where the x and y on the right hand side are the xy ViewPlanecoordinates.)

Definition of term: ViewPlaneEmbededInViewSpace

Definition of term: ViewPlaneToViewPlaneEmbeddedInViewSpace

Definition of term: ViewPlane·sp

In order to have a unique mapping, fixing the value of D to 1 creates anembedding in which the embedded ViewPlane is tangent to the north poleof the ViewSphere. We will adopt this as a convention, which will allowfor a unique mapping from any point in ViewSpace to a point on the planez=1 in ViewSpace. We will call this plane theViewPlaneEmbededInViewSpace. (In this sense, it is analogous toViewSpaceVS, which, as mentioned before, could have been called“ViewSphereEmbededInViewSpace.”) We will use the character p as ashorthand for the ViewPlaneEmbededInViewSpace space. With thisdefinition, we can define the mapping that goes from the ScreenSurfaceViewPlane (when D=1) back to ViewPlaneEmbededInViewSpace:ViewPlaneToViewPlaneEmbeddedInViewSpace, or with our short namingconventions, sp, which for clarity we will usually prepend the name ofthe ScreenSurface to: ViewPlane·sp.

So ViewPlane·sp, for the mapping to ViewPlaneEmbededInViewSpace from theuv Cartesian coordinates centered at the lower left of the ScreenSurfaceis:

$\begin{matrix}{x = {{{ViewPlane}.{sp}.{x\left\lbrack {u,v} \right\rbrack}} = {\frac{u - {{SW}/2}}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}}}} & (75) \\{y = {{{ViewPlane}.{sp}.{y\left\lbrack {u,v} \right\rbrack}} = {\frac{v - {{SW}/2}}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}}}} & (76) \\{z = {{{ViewPlane}.{sc}.{z\left\lbrack {u,v} \right\rbrack}} = 1}} & (77)\end{matrix}$

and then the mapping ViewPlane·sp from the xy Cartesian coordinatescentered at the center of the ScreenSurface:

$\begin{matrix}{x = {{{ViewPlane}.{sp}.{x\left\lbrack {x,y} \right\rbrack}} = {\frac{x}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}}}} & (78) \\{y = {{{ViewPlane}.{sp}.{y\left\lbrack {x,y} \right\rbrack}} = {\frac{y}{{SW}/2} \cdot {\tan\left\lbrack {{FOV}/2} \right\rbrack}}}} & (79) \\{z = {{{ViewPlane}.{sp}.{z\left\lbrack {x,y} \right\rbrack}} = 1}} & (80)\end{matrix}$

and then the mapping ViewPlane·sp from the (normal) radius angle polarcoordinates of the ScreenSurface:

$\begin{matrix}{x = {{{ViewPlane}.{sp}.{x\left\lbrack {{radius},{angle}} \right\rbrack}} = {\frac{radius}{{SW}/2} \cdot {\cos\lbrack{angle}\rbrack}}}} & (81) \\{y = {{{ViewPlane}.{sc}.{y\left\lbrack {{radius},{angle}} \right\rbrack}} = {\frac{radius}{{SW}/2} \cdot {\sin\lbrack{angle}\rbrack}}}} & (82) \\{z = {{{ViewPlane}.{sc}.{z\left\lbrack {{radius},{angle}} \right\rbrack}} = 1}} & (83)\end{matrix}$

The first two sets of equations define a rectangular portion of theplane ViewPlaneEmbededInViewSpace, the third a circular portion.

Definition of term: ViewSpaceToViewPlaneEmbededInViewSpace

Definition of term: ViewPlane·cp

We can now state the traditional 3D perspective mapping equation(independent of how many pixels there are) as the mappingViewSpaceToViewPlaneEmbededInViewSpace, short name ViewPlane·cp:

$\begin{matrix}{{x = {{{ViewPlane}.{cp}.{x\left\lbrack {X,Y,Z} \right\rbrack}} = \frac{X}{Z}}}{{y = {{{ViewPlane}.{cp}.{x\left\lbrack {X,Y,Z} \right\rbrack}} = \frac{Y}{Z}}},{{{ViewPlane}.{cp}.{z\left\lbrack {X,Y,Z} \right\rbrack}} = 1}}} & (84)\end{matrix}$

As with the cv mapping, because of the destructive nature of the cpmapping, no unique inverse for cp (pc) exists. But also as withViewSpaceVS, points in ViewPlaneEmbededInViewSpace are also points inViewSpace.

Definition of term: ViewPlaneEmbededInViewSpaceToVisualCoordinates

Definition of term: ViewPlane·pz

Since ViewPlaneEmbededInViewSpace is a subset of ViewSpace, the mappingfor ViewPlaneEmbededInViewSpaceToVisualCoordinates, short nameViewPlane·pz, is the same as that for ViewPlane·cz:

$\begin{matrix}{\phi = {{{ViewPlane}.{pz}.{\phi\left\lbrack {x,y,z} \right\rbrack}} = {{{ViewPlane}.{cz}.{\phi\left\lbrack {x,y,z} \right\rbrack}} = {a\;\tan\;{2\left\lbrack {y,x} \right\rbrack}}}}} & (85) \\{\theta = {{{ViewPlane}.{pz}.{\theta\left\lbrack {x,y,z} \right\rbrack}} = {{{ViewPlane}.{cz}.{\theta\left\lbrack {x,y,z} \right\rbrack}} = {\cos^{- 1}\left\lbrack \frac{z}{\sqrt{x^{2} + y^{2} + z^{2}}} \right\rbrack}}}} & (86)\end{matrix}$Converting from ViewPlane (ScreenSurface) Coordinates to its Equivalentin VisualCoordinates

This sub-section will derive an equation for converting a point given inViewSpace coordinates to (the polar form of) ViewPlaneEmbededInViewSpacecoordinates. An illustration of the relationship of the geometricelements involves is shown in FIG. 92.

Given: the positive z axis of ViewSpace is 9210, the ViewPlane is 9230,P0 9250 is the original point in xyz ViewSpace coordinates, P1 9260 isthe projection of P0 onto the ViewSphere in xyz ViewSpaceVS coordinates,θ 9240 is the eccentricity of P1 in VisualCoordinates, P2 9270 is theprojection of P0 onto the ViewPlaneEmbededInViewSpace, in radius, angleViewPlaneEmbededInViewSpace coordinates (not ViewPlane), and r 9220 isthe radius of the origin centered sphere that passes through P2, and P3is P2 in radius, angle ViewPlane coordinates (notViewPlaneEmbededInViewSpace).

Then:

$\begin{matrix}{\mspace{79mu}{\theta = {{{cz}.{\theta\left\lbrack {P\; 0} \right\rbrack}} = {\cos^{- 1}\left\lbrack \frac{P\; 0.z}{\sqrt{{P\; 0.x^{2}} + {P\; 0.y^{2}} + {P\; 0.z^{2}}}} \right\rbrack}}}} & (87) \\{\mspace{79mu}{\phi = {{{cz}.{\phi\left\lbrack {P\; 0} \right\rbrack}} = {a\;\tan\;{2\left\lbrack {{P\; 0.y},{P\; 0.x}} \right\rbrack}}}}} & (88) \\{\mspace{79mu}{{P\; 1.z} = {{{zv}.{z\left\lbrack {\phi,\theta} \right\rbrack}} = {\cos\lbrack\theta\rbrack}}}} & (89) \\{\mspace{79mu}{{P\; 1.{radius}} = {\sqrt{{{zv}.{x\left\lbrack {\phi,\theta} \right\rbrack}^{2}} + {{zv}.{y\left\lbrack {\phi,\theta} \right\rbrack}^{2}}} = {\sin\lbrack\theta\rbrack}}}} & (90) \\{\mspace{79mu}{r = {{{1/P}\; 1.z} = {1/{\cos\lbrack\theta\rbrack}}}}} & (91) \\{\mspace{79mu}{{P\; 2.{radius}} = {{r \cdot {\sin\lbrack\theta\rbrack}} = {{\frac{1}{\cos\lbrack\theta\rbrack} \cdot {\sin\lbrack\theta\rbrack}} = {\tan\lbrack\theta\rbrack}}}}} & (92) \\{\mspace{79mu}{{P\; 2.{radius}} = {\tan\left\lbrack {\cos^{- 1}\left\lbrack \frac{P\; 0.z}{\sqrt{{P\; 0.x^{2}} + {P\; 0.y^{2}} + {P\; 0.z^{2}}}} \right\rbrack} \right\rbrack}}} & (93) \\{\mspace{79mu}{{P\; 3.{angle}} = {{P\; 2.{angle}} = {{{cz}.{\phi\left\lbrack {P\; 0} \right\rbrack}} = {a\;\tan\;{2\left\lbrack {{P\; 0.y},{P\; 0.x}} \right\rbrack}}}}}} & (94) \\\begin{matrix}{{P\; 3.{radius}} = {{{\frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot P}\; 2.{radius}} = {\frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot {\tan\lbrack\theta\rbrack}}}} \\{= {\frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot {\tan\left\lbrack {\cos^{- 1}\left\lbrack \frac{P\; 0.z}{\sqrt{{P\; 0.x^{2}} + {P\; 0.y^{2}} + {P\; 0.z^{2}}}} \right\rbrack} \right\rbrack}}}\end{matrix} & (95)\end{matrix}$

Restating this, when the ScreenSurface is the ViewPlane, the cs mappingfrom ViewSpace to radius angle polar ViewPlane coordinates is:

$\begin{matrix}{{{cs}.{{radius}\left\lbrack {x,y,z} \right\rbrack}} = {\frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot {\tan\left\lbrack {\cos^{- 1}\left\lbrack \frac{P\; 0.z}{\sqrt{{P\; 0.x^{2}} + {P\; 0.y^{2}} + {P\; 0.z^{2}}}} \right\rbrack} \right\rbrack}}} & (96) \\{\mspace{79mu}{{{cs}.{{angle}\left\lbrack {x,y,z} \right\rbrack}} = {a\;\tan\;{2\left\lbrack {y,x} \right\rbrack}}}} & (97)\end{matrix}$

Restating this from VisualCoordinates:

$\begin{matrix}{{{cs}.{{radius}\left\lbrack {\phi,\theta} \right\rbrack}} = {\frac{{SW}/2}{\tan\left\lbrack {{FOV}/2} \right\rbrack} \cdot {\tan\lbrack\theta\rbrack}}} & (98) \\{{{cs}.{{angle}\left\lbrack {\phi,\theta} \right\rbrack}} = \phi} & (99)\end{matrix}$Comments on the ViewPlane

A previously very important reason that the traditional ViewPlane hasbeen so extensively used in software and effectively all hardware 3Dgraphics systems is the property that straight lines in ViewSpacetransform into straight lines in 2D ViewPlane coordinates. Any shading,lighting, texturing etc. on the line would not transform linearly, butthat isn't a concern when the line display “color” is just black orglowing green. What was a benefit is that transformed 3D lines andtriangles can be rendered in ViewSpace with (appropriate) 2D line andtriangle drawing algorithms, and if pixels need to be Z-buffered (e.g.,triangles are not pre-sorted in painter's order or the equivalent), thenvalues proportional to 1/w can be linearly interpolated to produce a “z”value to be z-buffered. This had the advantage of avoiding what wereexpensive division operations per pixel drawn, though multiply perViewSpace interpolated values (color, texture address(es), etc.) arestill required. However, now that complete complex shaders are executedat least once per drawn pixel, the division saving is no longerrelevant. The “last straw” will come when all geometric primitives arealways evaluated at least once per pixel. As of that point the onlyoperation that still might be performed in 2D is the interpolation ofcolor and z from micro-triangle vertices to perturbed sub-pixel samplinglocations. As mentioned earlier, the graphics pipeline described in thissystem assumes that all geometric primitives are sub-divided or theequivalent until they fall below a specified maximum size relative tothe local variable resolution pixel size.

III.E. Re-Definition of Resolution

To define what we will mean by variable resolution, we first must havean appropriate definition of resolution. With the concept of theViewSphere established, we can now formally define linear preceptorialresolution, or just resolution:

Definition of term: linear preceptorial resolution

Definition of term: resolution

Definition of term: visual spatial frequency

The standard definition of visual spatial frequency will be used as thedefinition of linear perceptual resolution, or just resolution. This isdefined as the spatial frequency in cycles per radian with a halfwavelength corresponding to a given perceptual distance on theViewSphere. The perceptual distance between two points on the surface ofthe ViewSphere is their “angular” distance on the great circleconnecting the two points. Thus all resolution measurements are made byin some way mapping two points to the surface of the ViewSphere. Table 1expresses common examples of resolution in more convenient units ofcycles per degree.

TABLE 1 Resolution examples. Example Cycles/Degree 20/10 rare bestvision 60 20/20 typical vision 30 20/30 frequently used vision 20 2Kdigital cinema 30 4K IMAX ™ 25 1920 × 1080 HDTV 30 1280 × 720 HDTV 20Sony ™ HMZ-T1 HMD 12The Resolution Equation

Given our definition of perceptual resolution, we are now in a positionto extend the definition to the concept of variable resolution.

The perceptual resolution of a point on the ScreenSurface in thedirection d is defined as the frequency of the corresponding perceptualdistance on the ViewSphere: our directional magnitude derivative of sv:

$\begin{matrix}{{rez} = \frac{1}{2 \cdot {\nabla_{d}{sv}}}} & (100)\end{matrix}$

This is the resolution equation. The resolution equation tells us the“resolution at a pixel.” Because this function will not necessary havethe same value in both the horizontal and vertical unit pixel directions(u and v), we cannot assume even approximately square pixels on theViewSphere. In fact, we will have to fight a bit to get them.

Note that this definition of resolution is based on the assumption thatthe highest perceivable spatial frequency is that given by the Nyquistlimit of one cycle per two pixels. This will be modified in a latersection.

Note that the this variable resolution equation gives the resolutionrelative to a point, not the resolution of a pixel edge or diagonal, asthe distance is actually measured in the tangent space. This is the mostcorrect symbolic equation. But sometimes the del operator can't beeasily applied, for example, because the mapping is a table. In thiscase an accurate numerical alternative is available. Assume that youwant to compute the resolution subtended by two points P₀ and P₁ on theScreenSurface. First use the sv mapping to obtain the two correspondingViewSpace points on the surface of the ViewSphere. These points bydefinition also represent normal vectors. This allows the great circledistance between them to be computed. Thus the corresponding resolutionis:

$\begin{matrix}{{rez} = \frac{1}{2 \cdot {\cos^{- 1}\left\lbrack {{{sv}\left\lbrack P_{0} \right\rbrack} \cdot {{sv}\left\lbrack P_{1} \right\rbrack}} \right\rbrack}}} & (101)\end{matrix}$

This is the discreet resolution equation. The points P₀ and P₁ couldrepresent two corners of a pixel, i.e. their difference is the unitvector u or v or the non-unit vector u+v. This last case is worthnoting. The resolution function has its lowest value 1/(2·|∇sv|), e.g.the longest spatial wavelength, in the direction of ∇sv. But this is fora radius one pixel distance. The lowest resolutions for square pixeltiling's of the ScreenSurface are effectively always found by measuringone of their diagonals, e.g. in the direction u+v or u−v, when theresults have been properly scaled by the path length √{square root over(2)}. Proponents of rectangular tilings like to point out that the humanvisual system is not as sensitive to 45° features. How large this effectis, is not the point. But, for example, when variable resolution isapplied to contact lens displays, generally the mappings arerotationally symmetric (e.g. diagonals of pixels will appear at allorientations around the display, including horizontal and vertical).This is an area where proponents of hexagonal tiling's get support fortheir point of view, because the hexagonal maximum wavelength is lowerrelative to that of a square pixel of the same area. The lowestresolution of a hexagon per unit area is higher than that of a square,for example, 30% fewer hexagons are needed to tile a region of the planethan squares for a given desired minimum resolution. Both the cones andthe receptor fields of the eye use six-way symmetry. Note that latermentions of “unity aspect ratio” have to do with equal directionalderivatives, not the specific pixel tiling.

Orthogonal Longitude Eccentricity Mappings

Definition of term: OrthogonalLongitudeEccentricity

We start with the case where sz·ϕ[u, v] is purely a function of u, andsz·θ[u, v] is purely a function of v: OrthogonalLongitudeEccentricitymappings. This simply means that the mapping equations for longitude andthe eccentricity are independent of each other: the equations areorthogonal. It is very important to note that this class of mappingspreserves the pixel structure of any Pixel Space of the surfaceassociated with the surface.

Linearly Longitudinal Mappings

Definition of term: LinearlyLongitudinal

Next we consider the case of an OrthogonalLongitudeEccentricity mappingwith the additional constraints that sz·ϕ[u, v] is a fixed linearfunction of u: LinearlyLongitudinal mappings. This just means that themapping equation for u is linear in longitude. Thus zs·ϕ and its inversesz·θ must be:

$\begin{matrix}{u = {{{zs}.{u\left\lbrack {\phi,\theta} \right\rbrack}} = {\phi \cdot \frac{SW}{2\pi}}}} & (102) \\{\phi = {{{sz}.{\phi\left\lbrack {u,v} \right\rbrack}} = {u \cdot \frac{2\pi}{SW}}}} & (103)\end{matrix}$

This component of the mapping distributes SW pixels equally around the2π radians of longitude, regardless of the eccentricity.

Because the circle of points with a constant ϕ (but all possible valuesof θ) on the ViewSphere is always a great circle, and thus has a radiusof 1 and a circumference of 2π, we have ∇_(v)sv=∇_(v)sz, and byinversion, also ∇_(θ)vs=∇_(θ)zs. However the circle of points with aconstant θ (but all possible values of ϕ) on the ViewSphere is not agreat circle (except the single case of θ=π), but instead has a radiusof sin [θ] and a circumference of 2π·sin [θ]. This means that∇_(θ)sv=sin [θ]·∇_(u)sz Taking this into account, for the perceptualdistance in the direction u we have:

$\begin{matrix}{{\nabla_{u}{sv}} = {{\sin\lbrack\theta\rbrack} \cdot \frac{2\pi}{SW}}} & (104)\end{matrix}$

and by equation (52) for the inverse, which is linear pixel density(pixels per radian):

$\begin{matrix}{{\nabla_{\phi}{vs}} = {\frac{1}{\sin\lbrack\theta\rbrack} \cdot \frac{SW}{2\;\pi}}} & (105)\end{matrix}$

What about ∇_(u)sv? It still can be any general function of θ. In thisLinearlyLongitudinal case it dynamically controls the aspect ratio ofthe approximately rectangular region of the ViewSphere that squareregions of the ScreenSurface map onto.

$\begin{matrix}{{AspectRatio} = {\frac{\nabla_{u}{sv}}{\nabla_{v}{sv}} = {\frac{\sin\lbrack\theta\rbrack}{\nabla_{v}{sv}} \cdot \frac{2\pi}{SW}}}} & (106)\end{matrix}$

In this way the resolution can be set in the v direction to whatever isdesired, but in the u direction it must follow equation (103).

When we do have an equation for ∇_(u)sv, by equation (52) we also havethe equation for ∇_(θ)vs, we can then obtain the equation for v simplyby integration:v=vs·v[θ]=∫_(θmin) ^(θ)∇_(θ′) vs·dθ′  (107)

Where θ_(min) is defined next.

Definition of term: θ_(min)

Definition of term: θ_(max)

θ_(min) is smallest value of θ used for a particularLinearlyLongitudinal mapping, θ_(max) is largest. In many casesθ_(max)=FOV/2. The chart (as described in VisualCoordinates) associatedwith a particular LinearlyLongitudinal mapping many times only includesthe region of the ViewSphere defined by values of e such thatθ_(min)≤θ<θ_(max). For portions of the ViewSphere outside this region,mappings from other charts in the overall atlas of charts have to beused instead of the LinearlyLongitudinal mapping. In most cases, θ_(max)represents the outer edge of the atlas of mappings, so no additionalchart (and associated mapping) for that portion of the ViewSphere needbe present. More details of what values θ_(min) may take on and why, andwhat sort of chart and associated mapping is used for this “north pole”region will be covered later in the discussion on end-caps.

Equations (102) through (107) describe all LinearlyLongitudinalmappings.

Locally Uniform Resolution

Now we will consider LinearlyLongitudinal mappings with the additionalrestriction that the mappings are locally uniform, and therefore so isthe resolution, e.g. square regions on the ScreenSurface map to anapproximately square regions on the ViewSphere: shape is locallypreserved. Locally uniform resolution mappings occurs when the localvalue of resolution is invariant of the direction chosen:∇_(v) sv=∇ _(u) sv  (108)

Such mappings are a strict subset of LinearlyLongitudinal mappings. Thusequation (105) is now also the value for ∇_(θ)vs, the resolution in theθ direction varies as the inverse sine of the eccentricity:

$\begin{matrix}{{rez} = {\frac{1}{\sin\lbrack\theta\rbrack} \cdot \frac{SW}{4\;\pi}}} & (109)\end{matrix}$

Applying equation (107) to obtain the v component:

$\begin{matrix}\begin{matrix}{v = {{{vs}.{v\lbrack\theta\rbrack}} = {\frac{SW}{2\pi} \cdot {\int_{\theta\;\min}^{\theta}\ {{\frac{1}{\sin\left\lbrack \theta^{\prime} \right\rbrack} \cdot d}\;\theta^{\prime}}}}}} \\{= {\left( {\frac{SW}{2\pi} \cdot {\log_{e}\left\lbrack {\tan\left\lbrack \frac{\theta}{2} \right\rbrack} \right\rbrack}} \right) - {\frac{SW}{2\pi} \cdot {\log_{e}\left\lbrack {\tan\left\lbrack \frac{\theta}{2} \right\rbrack} \right\rbrack}}}}\end{matrix} & (110)\end{matrix}$

Definition of term: Locally Uniform Resolution

Definition of term: LUR

Combined with equation (102) we now have a complete sv mapping.Equations (102) and (110) comprise the locally uniform mapping (a namedScreenSurface) also referred to as LUR. Note that LUR mapping is unique;it is the only one that obeys all of the constraints. Several propertiesof this mapping including its uniqueness make it quite valuable for usein variable resolution systems. For example, equation (110) is the innerloop of perceptual resolution driven hardware renderers. And withappropriately set parameters of θ_(min) and SW, it also fits much of theobserved visual resolution capabilities of the human eye and visualsystem.

FIG. 55 shows this mapping both on the ScreenSurface, where it is justordinary square pixels element 5510, and on the standard orthographicViewPlane projection, where it is a log half angle polar mapping:uniform rotation and log_(e)[tan [θ/2]] mapping in radius element 5520.Locally, shapes and therefore orientations on the ScreenSurface arepreserved on the surface of the ViewSphere. Local preservation ofrelative lengths and angles between vectors directly follows from this.

By combining in the standard mapping from ViewSpace toVisualCoordinates, cz, we can obtain the complete transform fromViewSpace to the ScreenSurface for the LocallyUniformResolution mapping:

$\begin{matrix}{u = {{{cz}.{u\left\lbrack {x,y,z} \right\rbrack}} = {{atan}\;{{2\left\lbrack {y,x} \right\rbrack} \cdot \frac{SW}{2\pi}}}}} & (111) \\\begin{matrix}{v = {{cz}.{v\left\lbrack {x,y,z} \right\rbrack}}} \\{= {{{\log_{e}\left\lbrack {\tan\left\lbrack \frac{\cos^{- 1}\left\lbrack \frac{z}{\sqrt{x^{2} + y^{2} + z^{2}}} \right\rbrack}{2} \right\rbrack} \right\rbrack} \cdot \frac{SW}{2}} - {{\log_{e}\left\lbrack {\tan\left\lbrack \frac{\theta_{\min}}{2} \right\rbrack} \right\rbrack} \cdot \frac{SW}{2\pi}}}}\end{matrix} & (112)\end{matrix}$

This can be simplified by collecting up the second constant term asv_(min):

$\begin{matrix}{v_{\min} = {{\log_{e}\left\lbrack {\tan\left\lbrack \frac{\theta_{\min}}{2} \right\rbrack} \right\rbrack} \cdot \frac{SW}{2\pi}}} & (113)\end{matrix}$

We can now restate cz as:

$\begin{matrix}{\mspace{76mu}{u = {{{cz}.{u\left\lbrack {x,y,z} \right\rbrack}} = {{atan}\;{{2\left\lbrack {y,x} \right\rbrack} \cdot \frac{SW}{2\pi}}}}}} & (114) \\{v = {{{cz}.{v\left\lbrack {x,y,z} \right\rbrack}} = {{{\log_{e}\left\lbrack {\tan\left\lbrack \frac{\cos^{- 1}\left\lbrack \frac{z}{\sqrt{x^{2} + y^{2} + z^{2}}} \right\rbrack}{2} \right\rbrack} \right\rbrack} \cdot \frac{SW}{2\pi}} - v_{\min}}}} & (115)\end{matrix}$

Algorithm for Conversion to ScreenSpace

Traditional pipeline computation:1/Pv·z→wPv·x*w*SW/2+SW/2→Ps·u,Pv·y*w*SW/2+SH/2→Ps·v

Variable Resolution Pipeline with constants: Z_(max)=cos [θ_(min)],v_(min)=log [tan [[θ_(min)/2] ]*SW/2π, and D=SW/sin [θ_(min)]:

Computation:Pv·z/sqrt[Pv·x ² +Pv·y ² +Pv·z ²]→z′

If z′>Z_(max): // End capD*Pv·x→Ps·u,D*Pv·y→Ps·v

else // locally uniform resolution mappinga tan 2[Pv·y,Pv·x]*(SW/2π)→Ps·u,log [tan [a cos [z′/2]]]*(SW/2π)−v _(min) →Ps·v

From a hardware point of view, a key insight is that the function:log_(e)[tan [a cos [x]/2]] can be implemented as a single fast dedicatedhardware function unit.

Log-Polar Coordinates and Spatial Variant Resolution

Definition of term: LogPolar

Definition of term: SpatialVarientResolution

Two dimensional LogPolar coordinates are a variation of standard polarcoordinates in which the radius component is a function of the log_(e)[] of a scaling of the standard polar coordinates radius, while the polarcoordinates angle component of both are the same.SpatialVarientResolution is effectively the same concept.

A general introduction to LogPolar mappings of the ViewPlane is [Araujo,H, Dias J. M. 1996. “An introduction to the log-polar mapping,” inProceedings of the 2nd Workshop Cybernetic Vision, December 1996, pp.139-144.].

Especially when the parameter a (to be described below) is included, theLogPolar mapping has also been referred to as spatial variantresolution. The main reference is [Wallace 1994: Wallace, R. et al.1994. Space Variant Image Processing. International Journal of ComputerVision 13, 71-90.].

LogPolar/SpatialVarientResolution mappings have been of interest because(with the appropriate constants) they appear to closely match how visualinformation is physically mapped into the brain's visual cortex, andalso because they also have proved to be an efficient space to performcertain image processing tasks in. This class of spaces has severalproperties of interest, most of which are described in the tworeferences given above. One in common with most of those taught here isthat LogPolar mappings are LinearlyLongitudinal. This means that allpoints of a particular longitude in VisualCoordinates fall onto the samevertical line on the ScreenSurface. While LogPolar mappings are not(even locally) shape-preserving, the mappings can be said to be anglepreserving in the sense that differences in “angle” (visual longitude,see below) between two points are preserved by the mapping. Locally“shape” preserving would mean that angles between local vectors arepreserved. That is, if points A, B, and C are points in the originalimage quite close to each other, then the angle between the localvectors

and

(e.g. cos⁻¹[

·

]) should have the same value both before and after the mapping isapplied. This is not true for LogPolar mappings, but it is true for theLocallyUniformResolution mapping. Also, the LocallyUniformResolutionmapping preserves the ratio of the lengths of the vectors: the ratio |

|/|

| has the same value both before and after the mapping is applied; againthis does not hold for the LogPolar mappings.

In our terminology, the LogPolar mapping is a specific ScreenSurfacemapping. However, the way that LogPolar mappings have been defined inthe literature are not in the form of any of the mappings we have beenusing to define ScreenSurface mappings. That is, they are not given as amapping from a ScreenSurface to ViewSpace, or to ViewSpaceVS, or toVisualCoordinates. Instead, the standard definitions of the LogPolarmapping is given as a mapping from ViewPlane ScreenSurface centered xycoordinates to an un-normalized radius and angle:angle=a tan 2[ViewPlane·y,ViewPlane·x]  (116)radius=log_(e)[√{square root over (ViewPlane·x ²ViewPlane·y ²)}|]  (117)

The angle is in units of radians, not pixels, and the radius is in unitsof log_(e)[pixel distance in ViewPlane coordinates]. To fully definewhat is meant by the LogPolar mapping, we will first have to normalizethem using the values of LogPolar·SW, LogPolar·FOV, and ViewPlane·SW.

The FOV of the two spaces, however, is generally the same, though theVisualField of a LogPolar mapping is a circle cut out of the (squarish)ViewPlane mapping's ScreenSurface. In cases where the ScreenSurfacesurface is defined by the equation θ<FOV/2, we have θ_(max)=FOV/2.

Even though one is derived from the other, the LogPolar mapping is adifferent instance of a ScreenSurface than the ViewPlane mapping, and(effectively) they will always have their own different values of SW andSH. Thus, when necessary, we will differentiate them by explicitlyreferring to ViewPlane·SW, LogPolar·SW, and LogPolar·SH. Since themapping is defined on a circular sub-region of the ViewPlane, theutilized portion of the ViewPlane is a square region, and soViewPlane·SW and ViewPlane·SH can be considered the same.

By inspection, LogPolar is a LinearlyLongitudinal mapping, so we knowwhat the u (longitude) component is in various other spaces:

$\begin{matrix}{\mspace{76mu}{u = {{{LogPolar}.{cs}.{u\left\lbrack {X,Y,Z} \right\rbrack}} = {{\frac{{LogPolar}.{SW}}{2\pi} \cdot {atan}}\;{2\left\lbrack {Y,X} \right\rbrack}}}}} & (118) \\{u = {{{ViewPlaneToLogPolar}.{u\left\lbrack {x,y} \right\rbrack}} = {{\frac{{LogPolar}.{SW}}{2\pi} \cdot {atan}}\;{2\left\lbrack {y,x} \right\rbrack}}}} & (119) \\{\mspace{76mu}\begin{matrix}{u = {{ViewPlaneEmbeddedInViewSpaceToLogPolar}.{u\left\lbrack {x,y} \right\rbrack}}} \\{= {{\frac{{LogPolar}.{SW}}{2\pi} \cdot {atan}}\;{2\left\lbrack {y,x} \right\rbrack}}}\end{matrix}} & (120) \\{\mspace{76mu}{u = {{{LogPolar}.{zs}.{u\left\lbrack {\phi,\theta} \right\rbrack}} = {\frac{{LogPolar}.{SW}}{2\pi} \cdot \phi}}}} & (121)\end{matrix}$

The v component will be a little harder to obtain.

Since the log_(e)[ ] returns negative numbers for inputs less than 1,most LogPolar mappings define the circular region of the ViewPlane witha radius less than 1 to be outside the main LogPolar chart. The minimumLogPolar·v coordinate value of 0 is defined to be at this radius. Thisis equivalent to the use of a θ_(min) value set to the eccentricitysubtended by a one pixel radius in the ViewPlane:

$\begin{matrix}{\theta_{\min} = {\tan^{- 1}\left\lbrack \frac{\tan\left\lbrack {{FOV}\text{/}2} \right\rbrack}{{{Viewplane}.{SW}}\text{/}2} \right\rbrack}} & (122)\end{matrix}$

The problem with this definition of θ_(min) is that the shape of theLogPolar mapping is dependent on both the number of pixels in theViewPlane ScreenSurface as well as the FOV. This means that changing thenumber of pixels doesn't just change the sampling density, it alsochanges the shape of the radius mapping function. And changing the FOVdoesn't just change the extent of the VisualField and the samplingdensity, it also changes the shape of the radius mapping function. Whilemuch of the existing literature defines the LogPolar mapping in thisshifting way, we will normalize our definition of the LogPolar mappingby taking θ_(min) as a fixed constant, and not dependent on either theViewPlane·SW or the FOV.

Past work apparently has been able to accept the shape change because ofthe use of the log_(e)[ ] function means that the change in shape of themapping is just a shift along the radius axis, and the shift is quitesmall for even large scale changes: a scale factor of two will onlyoffset the destinations of points into the LogPolar representation byless than a pixel in radius (as log_(e)[2]≈0.7). But when arguing overdifferent LogPolar fits of the visual cortex, people are concerned aboutsuch small offsets and small differences in the value of a (to bedescribed).) The equation above can be used to convert the assumptionsof other work into our normalized form.

We can convert from the radial distance in ViewPlane coordinates to afunction of visual eccentricity via the previously developed equation(but only for the non-normalized version of θ_(min) defined above):

$\begin{matrix}{{{ViewPlane}.{zs}.{{radius}\left\lbrack {\phi,\theta} \right\rbrack}} = {{\frac{{{ViewPlane}.{SW}}\text{/}2}{{tab}\left\lbrack {{FOV}\text{/}2} \right\rbrack} \cdot {\tan\lbrack\theta\rbrack}} = \frac{\tan\lbrack\theta\rbrack}{\tan\left\lbrack \theta_{\min} \right\rbrack}}} & (123)\end{matrix}$

We know that the final form of the equation for the v component of theLogPolar mapping will be of the form:

$\begin{matrix}{v = {{{LogPolar}.{zs}.{v\left\lbrack {\phi,\theta} \right\rbrack}} = {k\;{1 \cdot {\log_{e}\left\lbrack {k\;{2 \cdot \frac{\tan\lbrack\theta\rbrack}{\tan\left\lbrack \theta_{\min} \right\rbrack}}} \right\rbrack}}}}} & (124)\end{matrix}$

Where k1 and k2 are constants to be determined.

We know we want LogPolar·zs·v[θ_(min)]=0, so k2 must be a value thatsets the expression inside the log_(e)[ ] function to be 1 when0=θ_(min), which happens when k2=1, so k2 vanishes. This leaves us with:

$\begin{matrix}{v = {{{LogPolar}.{zs}.{v\left\lbrack {\phi,\theta} \right\rbrack}} = {k\;{1 \cdot {\log_{e}\left\lbrack \frac{\tan\lbrack\theta\rbrack}{\tan\left\lbrack \theta_{\min} \right\rbrack} \right\rbrack}}}}} & (125)\end{matrix}$

k1 is harder. We would like to keep ∇_(θ)LogPolar·sv·v as close to∇_(ϕ)LogPolar·sv·u as possible, so let's take a look at them:

$\begin{matrix}{{{\nabla_{e}{LogPolar}}.{sv}.v} = {{\frac{d}{dv} \cdot {{LogPolar}.{sv}.v}} = {{\frac{2}{\sin\left\lbrack {2 \cdot \theta} \right\rbrack} \cdot k}\; 1}}} & (126)\end{matrix}$

Since the LogPolar mapping is LinearlyLongitudinal, the scale change inthe direction is:

$\begin{matrix}{{{\nabla_{\phi}{LogPolar}}.{sv}.v} = {\frac{1}{\sin\lbrack\theta\rbrack} \cdot \frac{{LogPolar}.{SW}}{2\pi}}} & (127)\end{matrix}$

Here we see the fundamental difference between the LogPolar mapping andthe LocallyUniformResolution mapping. LocallyUniformResolution mappinghas, by definition, the same directional magnitude derivative in alldirections from a given point. The LogPolar mapping cannot be made to doso. In fact, the derivation of the unique LocallyUniformResolutionmapping proves that it is the only mapping that can have this property.

We can, however, come close, at least for small eccentricities. Forsmall values of θ, sin [θ]≈θ. So the following setting of k1 will make∇_(θ)LogPolar·sv·v≈∇_(ϕ)LogPolar·sv·u, for small values of θ:

$\begin{matrix}{{k\; 1} = \frac{{LogPolar}.{SW}}{2\pi}} & (128)\end{matrix}$so now we have:

$\begin{matrix}{v = {{{LogPolar}.{zs}.{v\left\lbrack {\phi,\theta} \right\rbrack}} = {\frac{{LogPolar}.{SW}}{2\pi} \cdot {\quad{\left( {{\log_{e}\left\lbrack {\tan\lbrack\theta\rbrack} \right\rbrack} - {\log_{e}\left\lbrack {\tan\left\lbrack \theta_{\min} \right\rbrack} \right\rbrack}} \right)\;{{and}:}}}}}} & (129) \\\begin{matrix}{{{\nabla_{e}{LogPolar}}.{sv}.v} = {{\frac{2}{\sin\left\lbrack {2 \cdot \theta} \right\rbrack} \cdot \frac{{LogPolar}.{SW}}{2\pi}} = {\frac{1}{\cos\lbrack\theta\rbrack} \cdot \frac{1}{\sin\lbrack\theta\rbrack} \cdot \frac{{{LogPolar}.{SW}}\text{/}2}{2\pi}}}} \\{= \frac{{\nabla_{\phi}{LogPolar}}.{sv}.v}{\cos\lbrack\theta\rbrack}}\end{matrix} & (130)\end{matrix}$

Unfortunately, the scale in the θ direction will differ from that in ϕdirection as θ gets larger, e.g. by 1/cos [θ]. For (relatively) smallFOVs, this isn't off by much, as a FOV of 60° will have a maximum visualeccentricity of 30°, and 1/cos [30° ]≈1.15. This isn't too bad,especially compared to other factors that have to be taken intoconsideration when deeply modeling the optics of the eye, such asconverting from visual eccentricity to retinal eccentricity. But by aFOV of 90°, the ratio raises to about 1.4, and goes completely of thechart as one approaches 180°: at a FOV of 170°, the ratio is up to 11.5!As the maximum visual eccentricity of the human can be over 105°, anyform of the LogPolar mapping becomes undefined.

Summarizing, we can now define:

Definition of term: LogPolar·zs

$\begin{matrix}{\mspace{76mu}{u = {{{LogPolar}.{zs}.{u\left\lbrack {\phi,\theta} \right\rbrack}} = {\frac{{LogPolar}.{SW}}{2\pi} \cdot \phi}}}} & (131) \\{v = {{{LogPolar}.{zs}.{v\left\lbrack {\phi,\theta} \right\rbrack}} = {{\frac{SW}{2} \cdot {\log_{e}\left\lbrack {\tan\lbrack\theta\rbrack} \right\rbrack}} - {\frac{SW}{2\pi} \cdot {\log_{e}\left\lbrack {\tan\left\lbrack \theta_{\min} \right\rbrack} \right\rbrack}}}}} & (132)\end{matrix}$

and derivatives in the ϕ and θ directions:

$\begin{matrix}{\mspace{76mu}{{{\nabla_{\phi}{LogPolar}}.{sv}.v} = {\frac{1}{\sin\lbrack\theta\rbrack} \cdot \frac{{LogPolar}.{SW}}{2\pi}}}} & (133) \\{{{\nabla_{\theta}{LogPolar}}.{sv}.v} = {{\frac{1}{\cos\lbrack\theta\rbrack} \cdot \frac{1}{\sin\lbrack\theta\rbrack} \cdot \frac{{LogPolar}.{SW}}{2\pi}} = \frac{{\nabla_{\phi}{LogPolar}}.{sv}.v}{\cos\lbrack\theta\rbrack}}} & (134)\end{matrix}$

The inverses of these particular mappings and mappings to and from otherknown coordinate systems can be easily derived using the previous spaceconversion mappings. Though, as described, the mappings to and from theViewPlane are problematic for the reasons mentioned.

Definition of term: LogPolar·SH

Note that LogPolar·SH is defined by:

$\begin{matrix}\begin{matrix}{{{LogPolar}.{SH}} = {{\frac{SW}{2\pi} \cdot {\log_{e}\left\lbrack {\tan\left\lbrack \theta_{\max} \right\rbrack} \right\rbrack}} - {\frac{SW}{2\pi} \cdot {\log_{e}\left\lbrack {\tan\left\lbrack \theta_{\min} \right\rbrack} \right\rbrack}}}} \\{= {\frac{SW}{2\pi} \cdot {\log_{e}\left\lbrack \frac{\tan\left\lbrack \theta_{\max} \right\rbrack}{\tan\left\lbrack \theta_{\min} \right\rbrack} \right\rbrack}}}\end{matrix} & (135)\end{matrix}$

In the past, when attempts were made to fit the LogPolar mapping toknown data about the human eye and visual system resolution as afunction of visual eccentricity, different k1 constants have been used,different values of θ_(min) (usually 1° or a little more) are used, andoften an offset a is added inside the log_(e)[ ] term, which, whengreater than 1, also eliminates the undefined radius 1 hole in themapping. The latter is useful when building physical lens basedimplementations of the mapping. So we have a modified radius function:radius=log_(e)[√{square root over (ViewPlane·x ²ViewPlane·y²)}+a]  (136)

This is back in the un-normalized form, because this is the space inwhich a is usually defined. This form is harder to analyze, as a closedform of the derivative doesn't exist.

The methods described in this document are more general, and have beenreferred to as variable resolution. The phrase “spatial variantresolution” is more specific than that of “variable resolution,” in thatit tells one in what way does the resolution vary (by space). But aspresently the term “space variant resolution” is tightly identified withLogPolar representations, in this document the more general techniqueswill continue to be referred to as variable resolution.

The previously mentioned match of LogPolar representations to the humanvisual system are for areas outside the central foveal region, a regionapproximately two degrees of visual angle across (e.g., all of theretina inside about one degree of eccentricity). Thus many existingmodels that use LogPolar mapping elsewhere switch to a more constanthigh resolution mapping for the foveal region; thus the full modelconsists of a chart of two maps: a LogPolar mapping outside of onedegree of eccentricity, and a constant density EndCap mapping inside onedegree of eccentricity. The actual cut point between the two maps isgenerally parameterized by a constant that is effectively the same asthe θ_(min) constant. But as previously mentioned, when the applicationis to perform the LogPolar mapping optically, it is usually easier tonot use an end-cap, but a value of the parameter a larger than 1.

Comparing the Log-Polar Mapping with the Locally Uniform ResolutionMapping

In the last section we have already compared many aspects of theLogPolar mapping with LocallyUniformResolution. Both areLinearlyLongitudinal mappings, so they have the same equation for zs·uand ∇_(θ)LogPolar·sv·u. But while the LocallyUniformResolution mappinghas the same equation for ∇_(θ)LocallyUniformResolution·sv·v as∇_(ϕ)LocallyUniformResolution·sv·u, which is why the mapping is locallyshape preserving (as well as locally visual angle preserving), theLogPolar mapping does not. The effect can be kept small for smalleccentricities, but blows up at larger ones, and is ill defined at orbeyond a FOV of 180°, while the LocallyUniformResolution mapping is welldefined for FOVs approaching 360° (even though at eccentricities above90° the resolution starts going up again!).

Since such a superior mapping exists, why was the LogPolar mapping everproposed in the first place, and why have so many researchers continuedto use it?

In all likelihood, the LogPolar mapping was first proposed because thoseinvolved were using the traditional flat projection plane common to mostmanmade optical systems: cameras with flat planes of film, and newerdigital cameras of flat light sensitive imaging chips. We still don'thave the technology to make doubly curved surface semi-conductor imagingor display chips. With film, there have been some exceptions. The mostnotable is probably the 200 inch Mount Palomar Telescope, which has aspherically curved film plate and specially produced spherically curvedfilm! However the reason for this was to reduce spherical aberrations ofthe telescope's optical system. The field of view is considerablynarrower than that of the eye.

If one is only familiar with the planer projection model, or is forcedto use it for technical reasons, then applying a log_(e) function to theradius of a polar coordinate representation of the ViewPlane seems likea natural step. The only still or video images most people can get areall produced with the planer projection model. It's the “if all you haveis a hammer, every problem looks like a nail” sort of situation.

Even when restricted to a planer image acquisition surface, there stillare tricks that can be played with the optics. The common example are“fisheye” lenses. (This is amusing, because the human eye is basically afish eye adapted to use in air, and has a wider field of view than mostany “fisheye” lens.) This name covers a number of different ways thatthe radius function can be manipulated optically. Some lenses have beenspecially designed so that their radius manipulation implements theViewPlane to LogPolar mapping optically, making optimal use of thepixels on the planer imaging device at the focal plane of the optics.This is one quite valid reason for using the LogPolar mapping: it can berelatively inexpensively be actually implemented with existingtechnology, and can result in considerable computational savings forsome image processing and computer vision tasks. The fact that it mightnot completely accurately model the human visual system's representationis irrelevant, and the slightly non-uniform local resolution of themapping can be worked around.

There are less excuses when it comes to modeling the human eye. It haslong been known that the eye has a spherical imaging surface—inanalyzing it, one should start with a spherical projection, like theViewSphere. But the basic mathematical techniques behind such mappingsare not widely understood, and mainly applied in a few niches, such ascomputer rendering for hemispherical dome displays, and some of theassociated video projection techniques. The combination of using aspherical projection and variable resolution appears just to never havecome up before. More details about how well the LocallyUniformResolutionmapping can fit the known properties of the human eye and visual systemwill be discussed in a later section.

Later several techniques for building practical LocallyUniformResolutionmapping based displays and cameras using today's existing technologicaltechniques will be discussed. Not too many years ago, not all of thesetechniques were practical, which could be another reason why the mappinghasn't been explored before.

It should be noted that being confined to using imaging devices on theViewPlane is not a legitimate impediment to using theLocallyUniformResolution mapping rather than the LogPolar mapping. Belowis the definition of the LocallyUniformResolution mapping in terms ofViewPlane coordinates:

Definition of term: LocallyUniformResolution·ps

$\begin{matrix}\begin{matrix}{u = {{LocallyUniformResolution}.{ps}.{u\left\lbrack {{{ViewPlane}.x},{{ViewPlane}.y}} \right\rbrack}}} \\{= {{\frac{{LocallyUniformResolution}.{SW}}{2\pi} \cdot {atan}}\;{2\left\lbrack {{{ViewPlane}.y},{{ViewPlane}.x}} \right\rbrack}}}\end{matrix} & (137) \\\begin{matrix}{v = {{LocallyUniformResolution}.{ps}.{v\left\lbrack {{{ViewPlane}.x},{{ViewPlane}.y}} \right\rbrack}}} \\{= {{\frac{SW}{2\pi} \cdot {\log_{e}\left\lbrack {\tan\left\lbrack {{\tan^{- 1}\left\lbrack {\frac{\sqrt{{{ViewPlane}.x^{2}} + {{ViewPlae}.y^{2}}}}{{SW}\text{/}2} \cdot {\tan\left\lbrack \frac{FOV}{2} \right\rbrack}} \right\rbrack}\text{/}2} \right\rbrack} \right\rbrack}} - {\frac{SW}{2\pi} \cdot {\log_{e}\left\lbrack {\tan\left\lbrack \theta_{\min} \right\rbrack} \right\rbrack}}}}\end{matrix} & (138)\end{matrix}$

As previously mentioned, this radius transform can also be implementedcompletely optically for existing planer image plane image capturedevices.

End Caps

There is a singularity present at v=0 where the resolution goes toinfinity in LinearlyLongitudinal mappings. Most real variable resolutionLinearlyLongitudinal mappings effectively switch to another mappingdependency on θ at about 1°. We have defined the end cap takeover angleas θ_(min). The alternative mapping most simply is just near constantresolution, or only varies by a small additional factor.

A very simple way to “implement” this is to add an additional chartcalled EndCap to the atlas of mappings for a particular namedScreenSurface (which, remember, is a manifold). The surface for thisEndCap chart is will be the set of points that the inverse of itsmapping function takes the set of all points on the ViewSphere with zvalues above cos [θ_(min)] to. The mapping function for this EndCapchart can be bound by defining a specific mapping of ScreenSurfacesurface to or from any of the well-defined coordinate systems. In theexample we will give here we will do this by defining the EndCap·vsmapping. (Note that this denotes a different vs mapping than the one ofthe primary chart of the ScreenSurface. EndCap is the name of the secondchart, it is not a named ScreenSurface, it is just a component of anyScreenSurface, e.g. the full name is <named ScreenSurface>·EndCap. Whenno chart name is specified (as we have been doing up till now), theconvention is that the primary (first) chart is being implicitlyreferenced.) There are a number of different mappings that can work wellfor endcaps, but the simplest is just an orthographic projection, whichhas the advantage of being simple to implement in hardware. Here wedefine:

$\begin{matrix}{{{EndCap}.{vs}.{u\left\lbrack {x,y,z} \right\rbrack}} = {x \cdot \frac{{EndCap}.{SW}}{\sin\left\lbrack \theta_{\min} \right\rbrack}}} & (139) \\{{{EndCap}.{vs}.{v\left\lbrack {x.y.z} \right\rbrack}} = {y \cdot \frac{{EndCap}.{SW}}{\sin\left\lbrack \theta_{\min} \right\rbrack}}} & (140)\end{matrix}$

Note that the ScreenWidth here is that of this second mapping, not thefirst, and here ScreenHeight equals ScreenWidth (this is a square patchover a round hole). Again the domain of this mapping in thisScreenSurface is only those points which map via sv to a z value greaterthan cos [θ_(min)]. The resolution of the primary mapping at thisboundary will be a constant when a LinearlyLongitudinal mapping is beingused. Thus to ensure first order continuity between the two charts, thevalue of EndCap·SW can be adjusted so that the EndCap mapping will havethe same resolution at the boundary. (Orthographic projection has aslightly varying resolution in the region over the north pole, but forany given eccentricity the resolution will be constant. We will not givethe equation to force the matching of the resolution of the two chartsat the boundary here.) Order one continuity in resolution at theboundary means that there will be order one continuity in both mappedpixel size and sample density. In practice, second order continuity atthe boundary can be achieved by special casing the sampling patterns inall the boundary pixels to smooth the changes

When modeling the eye's retina, the EndCap mapping is that of the fovea,which is quite different from the mapping outside the fovea. Also, thefoveal mapping actually varies by as much as a factor of three in peakresolution by the individual. Most contact lens' displays can't use suchvariable foveal resolution maps, because slight shifts in the alignmentof the contact lens on the cornea correspondingly shifts the peak fovealresolution point all about the foveal portion of the display and beyond.Instead, a constant high resolution map is required.

A feature of some mappings, in particular the LocallyUniformResolutionmapping, is that they are well defined past 90°. The only difference isthat the resolution can get higher again past 90°. This actually happenson the human eye, where portions of the visual field do reach as much as105°. There, the density of retinal cones does go up.

The end cap technique basically put a cap on the north pole. Mostmappings do not need to be defined anywhere up to 180°−θ_(min). But whenrequired, the singularity at the south pole can be capped as well, usingexactly the same methods. In the sequel, when such one or two caps areneeded, they are implicitly assumed to be defined, and will only becalled out as necessary. Also, there is nothing too special about 1°;for any given real mapping, generally a workable angle somewhere in therange of one half to ten degrees can be found for θ_(min). Again,besides the orthographic end cap mapping, many others can be defined,most with better continuity.

VisualCoordinates to Retinal ScreenSurfaces

Overview

This chapter introduces the concept of the RetinalSphere, which physicalspace for points on the surface of the physical retina. Additionalcoordinate systems and two new ScreenSurface manifolds defined on theRetinalSphere support the physical and visual modeling of retinal conesand retinal midget ganglion cells. But the most critical detail, thederivation of a specific primary mapping function for each of thesemanifolds, will be left to a still later section.

SUMMARY

This chapter first defines a new two dimensional physical space: theRetinalSphere, corresponding to the physical world points on the surfaceof the retina.

ViewSpaceRS will be defined as the embedding of the RetinalSphere intoViewSpace.

RetinalCoordinates will be defined as a coordinate frame forViewSpaceRS.

RetinalCones will be a new named ScreenSurface manifold defined formodeling retinal cones.

RetinalMidget will be a new named ScreenSurface manifold defined formodeling retinal midget ganglion cells.

The conversion from RetinalCoordinates to VisualCoordinates(RetinalCoordinatesToVisualCoordinates) cannot be expressed as a closedform equation, due to the non-linear effects of the eye's optics in theeccentricity direction. Instead, a table interpolation will have to beused.

If, later down the line, any specific mapping's definition involvesconcatenation with the table based RetinalCoordinatesToVisualCoordinatesconversion, then that mapping will have to be considered as table basedas well.

Defined ScreenSurface Spaces and their Mappings

Definition of term: RetinalSphere

The RetinalSphere is defined to be the two dimensional closed surface ofa sphere in ViewSpace, with a radius equal to the retinal radius(default 12 mm), centered at the origin. The RetinalSphere representsthe eye's physical retinal surface. No special character is defined forthis space, as it is just S2, a character name will be assigned to aparticular coordinate system defined on the surface of theRetinalSphere. (Even though the RetinalSphere as defined is a surface,we still will usually refer to “the surface of the RetinalSphere” toreinforce this point.)

The RetinalSphere is not just magnified version of the ViewSphere.Points on the surface of the ViewSphere represent the projection of anypoints that lie on a straight line in ViewSpace from the origin ofViewSpace through a given point on the ViewSphere to that same point.Points on the surface of the RetinalSphere represent the physicallocations in space of certain retinal neurons. The optics of the eyecause points in ViewSpace that project to a specific point on thesurface of the ViewSphere to be optically imaged onto a different,non-corresponding point on the surface of the RetinalSphere. Points onthe surface of the RetinalSphere correspond to where light is physicallyprojected by the cornea and the lens; points on the surface of theViewSphere correspond to the angles that the light would have come in atif there were no optics present and that the light would haveintersected (or nearly) the EyePoint. (The RetinalSphere is a particularformalization of what was defined anatomically as the retinal sphere.)

A three dimensional embedding of the RetinalSphere into threedimensional ViewSpace will be defined; see ViewSpaceRS. One existingstandard reference two dimensional coordinate system will be defined forthe surface of the RetinalSphere; see RetinalCoordinates.

The actual mapping caused by the eye's optics between points on thesurface of the ViewSphere to points on the surface of the RetinalSpherewill be defined later, in particular it will be in the form of a mappingfrom VisualCoordinates to RetinalCoordinates.

Definition of term: ViewSpaceRS

ViewSpaceRS is defined to be the set of three dimensional points inViewSpace that are the embedding of the two dimensional points of theRetinalSphere into three dimensional ViewSpace. (We could have calledthis space “RetinalSphereEmbededInViewSpace”, but for a shorter name, welet the “RS” after “ViewSpace” stands for “RetinalSphere”.) Theindividual coordinate components of ViewSpaceRS will be denoted by x, y,and z

Definition of term: RetinalCoordinates

Definition of term: r

RetinalCoordinates is a coordinate system for the surface of theRetinalSphere. It is akin to VisualCoordinates, in that both used thesame longitude eccentricity angular parameterizations of theirrespective sphere's surface. To differentiate the angular parameters forRetinalCoordinates from those of VisualCoordinates, the upper case Greekletters will be used: retinal longitude will be denoted by Φ, andretinal eccentricity will be denoted by Θ.

The shorthand name for the RetinalCoordinates is r. Because theconversion between VisualCoordinates and RetinalCoordinates is fairlywell defined, RetinalCoordinates (like VisualCoordinates) is not aScreenSurface (this is why it can have a separate shorthand name).

Definition of term: VisualCoordinatesToRetinalCoordinates

Definition of term: zr

While there is no (accurate) closed form equation relatingVisualCoordinates to RetinalCoordinates, any particular wide angleschematic eye can be raytraced to produce a numeric solution. Longitudeis preserved, so we have:

$\begin{matrix}\begin{matrix}{{{RetinalCoordinates}.\Phi} = \begin{matrix}{VisualCoordinatesTo} \\{{RetinalCoordinates}.{\Phi\left\lbrack {\phi,\theta} \right\rbrack}}\end{matrix}} \\{= \phi}\end{matrix} & (141)\end{matrix}$

But the equation for eccentricity is table based:

$\begin{matrix}\begin{matrix}{{{RetinalCoordinates}.\Theta} = \begin{matrix}{VisualCoordinates} \\{{ToRetinalCoordinates}.{\Theta\left\lbrack {\phi,\theta} \right\rbrack}}\end{matrix}} \\{= {{VisEccToRetEcc}\lbrack\theta\rbrack}}\end{matrix} & (142)\end{matrix}$

where the table VisEccToRetEcc[ ] will be defined in a later section.

The shorthand name for this mapping is zr.

Definition of term: RetiniCoordinatesToVisualCoordinates

Definition of term: rz

RetinalCoordinatesToVisualCoordinates is the inverse ofVisualCoordinatesToRetinalCoordinates. The shorthand name for thisconversion is rz. Again, longitude is preserved, so we have:

$\begin{matrix}\begin{matrix}{{{VisualCoordinates}.\phi} = \begin{matrix}{{RetinalC}\;{oordinatesTo}} \\{{VisualCoordinates}.{\phi\left\lbrack {\Phi,\Theta} \right\rbrack}}\end{matrix}} \\{= \Phi}\end{matrix} & (143)\end{matrix}$

But again the equation for eccentricity is table based:

$\begin{matrix}\begin{matrix}{{{VisualCoordinates}.\theta} = \begin{matrix}{RetinalCoordinatesTo} \\{{VisualCoordinates}.{\theta\left\lbrack {\Phi,\Theta} \right\rbrack}}\end{matrix}} \\{= {{RetEccToVisEcc}\lbrack\Theta\rbrack}}\end{matrix} & (144)\end{matrix}$

where the table RetEccToVisEcc[ ] will be defined in a later section.

Definition of term: RetinalCones

RetinalCones is a named ScreenSurface for the retinal cones. It is usedto map a uniform array of equal size cones on a 2D surface to theirphysical instances on the RetinalSphere, or other coordinate systems.

Definition of term: RetinalCones·sr

The mapping of RetinalCones to RetinalCoordinates is defined byRetinalCones·sr.

Definition of term: RetinalCones·rs

The mapping of RetinalCoordinates to RetinalCones is defined byRetinalCones·rs, which is the inverse of RetinalCones·sr.

Definition of term: RetinalCones·sz

The mapping of RetinalCones to VisualCoordinates is defined byRetinalCones·sz. This is defined by the concatenation of the mappingRetinalCones·sr with the conversionRetinalCoordinatesToVisualCoordinates.

Definition of term: RetinalCones·zs

The mapping of VisualCoordinates to RetinalCones is defined byRetinalCones·zs, which is the inverse of RetinalCones·sz.

Definition of term: RetinalMidget

RetinalMidget is a named ScreenSurface for the retinal midget ganglioncells. It is used to map a uniform array of equal size midget ganglioncells on a 2D surface to their physical instances on the RetinalSphere,or other coordinate systems.

Definition of term: RetinalMidget·sr

The mapping of RetinalMidget to RetinalCoordinates is defined byRetinalMidget·sr.

Definition of term: RetinalMidget·rs

The mapping of RetinalCoordinates to RetinalMidget is defined byRetinalMidget·rs, which is the inverse of RetinalMidget·sr.

Definition of term: RetinalMidget·sz

The mapping of RetinalMidget to VisualCoordinates is defined byRetinalMidget·sz. This is defined by the concatenation of the mappingRetinalMidget·sr with the conversionRetinalCoordinatesToVisualCoordinates.

Definition of term: RetinalMidget·zs

The mapping of VisualCoordinates to RetinalMidget is defined byRetinalMidget·zs, which is the inverse of RetinalMidget·sz.

IV. The Variable Resolution Nature of the Human Eye

The human eye, like digital cameras, is comprised of a large number ofdiscrete light catching “pixels,” called retinal photoreceptors. Theretina has two types of photoreceptors: the night-vision photoreceptiverods, and the daylight and indoor lighting sensing photoreceptive cones.While there are many more rods (approximately eighty to one hundredtwenty million) in the eye than cones (approximately five to sixmillion); the rods trade-off increased light sensitivity for lowerresolution, so much so that they provide less visual resolution than thecones do. So for the purposes of understanding the uppermost resolutionlimits of the eye, and how to optimally display to it, it is theresolution capabilities of the cones that must be understood.

In most man-made cameras, all the pixels are of the same size, andspaced the same amount apart from each other. But the cones of the humaneye vary quite a bit in size, and more importantly, the spacing betweenthem varies even more. Furthermore, the “pixels” that are carried outthe back of the eye by the optic nerve to the rest of the brainrepresent the results of retinal processing that groups the outputs ofmany cones together, making the “effective pixel size” even larger. Whatthis means is that the human eye does not have anywhere near uniformresolution across its field of view. In fact, the “area” of a pixel onthe optic nerve can vary by a factor of one thousand from the highestresolution portion of the eye (the center of the fovea) to the lowest(the far periphery). Putting it another way, while the eye has as manyas six million individual retinal cone photoreceptors, there are lessthan one third of a million optic nerve fibers for carrying “pixeldata.” If an eye mounted display (specifically including contact lensdisplays) can be engineered to vary the size of the discrete lightproducing pixels to match the resolution of the portion of the retinathat the particular pixel will display to, then there is the opportunityto only need similarly small numbers of them; and therefore potentiallyalso only the need to render similarly small numbers of them for use onthe display. This represents a considerable savings over the currentart.

These “pixels of the optic nerve” are the outputs of the retinal midgetganglion cells. Each such cell has an associated visual receptor field,with a field center input from one or more cone cells, and a largersurround input from (approximately) seven or (many) more cone cells. Itis the combined visual field of all the cones that contribute to thecenter field that effectively determines what you could call the “visualpixel size” that the retinal midget ganglion cells detect. To understandhow this size varies across the visual field of the human eye, severalother factors must be taken into account first. So we will brieflyidentify the relevant factors, followed by a detailed sub-section oneach factor. As we get into the details, we will finally start definingmost of the specific mapping equations for the retinal spaces.

The “visual pixel size” mostly varies with eccentricity. The variationin size is approximately circularly symmetric around the center of theretina; the smallest size (highest resolution) occurs inside the fovea,the largest size (lowest resolution) occurs at the far periphery.

The retinal image is magnified. Because of the optics of the eye, thevisual image is magnified on the surface of the retina. This means thatany given visual eccentricity maps to a greater retinal eccentricity, bya factor of as much as 1.38, though the actual amount of magnificationvaries, and is becomes somewhat lower at larger eccentricities. Bycontrast, the retinal longitudinal angle is the same as the visuallongitudinal angle. This is important, as it means that the retinalangle in the eccentricity direction caused by the local spacing betweencones will be larger than the corresponding visual angle. In otherwords, from the visual side the cones look slightly flattened(approximately an ellipse with a ratio of major to minor axes of 1.38).

Cones are not always “flat-on” to the local surface of the retina. Togather the maximum amount of light, cones actively point in thedirection of the center of the exit pupil of the eye as it appears fromtheir individual location on the surface of the retina, e.g. not thesame direction in general as the normal to the (local) surface of theretina. This cause the cones to look slightly “squashed” from the visualside, though not enough to cancel out the opposite effect caused by theretinal image magnification.

The retinal size of cones is governed by several functions. The retinalsize of the cones is governed within the fovea by one constant and onefunction, and outside the fovea grows larger with increased eccentricityby a different function, and then becomes constant again.

The spacing between cones is governed by two functions. Inside the subportion of the fovea with less than 0.7° visual eccentricity, the fovealrod-free zone, there are no rods, only cones, and therefore the spacingbetween cones is determined by the size of the cones. Outside the fovealrod-free zone, rods start appearing between the individual cones, andthe spacing between individual cones grows by a different function,which is greater than that determined by the size of the cones.

The number of cones that contribute to the center input of retinalmidget ganglion cells varies with eccentricity. Inside a visualeccentricity of 6°, each midget ganglion cell obtains its center inputfrom precisely one cone cell, so the visual pixel size of the midgetganglion cell is the same as that of the underlying cone cells. Outsidethis region, the center input to midget ganglion cells start to comefrom more than one cone cell, with the number of such cones increasingmore and more with increasing visual eccentricity.

Now we will go into more details on each of these factors.

“Visual Pixel Size” Mostly Varies with Eccentricity

To a first approximation, the resolution mapping of the human eye is anOrthogonalLongitudeEccentricity mapping. Actually, it is almost aLinearlyLongitudinal mapping. That is, the preceptorial resolution ofthe human eye drops off with increasing visual eccentricity in almostthe same way regardless along which longitude the drop off is measured.While there is no longitudinal difference in the drop off within thefoveal region, in fact, outside the fovea, the drop off is somewhatslower in the nasal direction and the temporal direction than it is inthe superior or oblique direction. The drop off in the nasal directionis slightly slower than in the temporal direction. The fact thatslightly higher resolution is preserved in the nasal quadrant is becausethat is where the stereo overlap between the two eyes is.

This same slightly perturbed longitudinally symmetry is also present inthe distribution of the size of cones, and the spacing between cones.

The Retinal Image is Magnified

The purpose of this sub-section is define the RetinalCoordinatesconversion to and from VisualCoordinates.

Because of the optics of the eye, retinal eccentricity is not the sameas visual eccentricity. Optical effects cause the visual image to bemagnified on the surface of the retina. This means that any given visualeccentricity maps to a greater retinal eccentricity. The amount ofmagnification is not a constant, and itself varies with visualeccentricity. There is no closed form equation for the exactrelationship, instead the magnification has to be found by explicit raytracing of an optical model of the eye. And because there is no one“standard” wide angle optical model of the human eye, there is nostandard conversion table; different authors end up using differentmagnifications. Fortunately, because of the symmetries of the eye,retinal longitude is the same as visual longitude.

In this document to have a consistent conversion from visualeccentricity to the corresponding retinal eccentricity (and thereverse), we will use the result of ray tracing the particular eye modelof [Deering, M. 2005. A Photon Accurate Model of the Human Eye. ACMTransactions on Graphics, 24, 3, 649-658]. In table 2 below theconversion factor is given for every one degree of visual eccentricitythat the corresponding retinal eccentricity is, followed by the currentmagnification factor.

TABLE 2 Table of retinal eccentricities corresponding to visualeccentricities. Visual Retinal Eccentricity Eccentricity Retinal/Visual0° 0.000° 1.380 1° 1.380° 1.380 2° 2.759° 1.379 3° 4.138° 1.379 4°5.517° 1.379 5° 6.895° 1.379 6° 8.273° 1.379 7° 9.650° 1.379 8° 11.026°1.378 9° 12.402° 1.378 10° 13.776° 1.378 11° 15.149° 1.377 12° 16.521°1.377 13° 17.892° 1.376 14° 19.261° 1.376 15° 20.628° 1.375 16° 21.994°1.375 17° 23.358° 1.374 18° 24.720° 1.373 19° 26.079° 1.373 20° 27.437°1.372 21° 28.792° 1.371 22° 30.145° 1.370 23° 31.495° 1.369 24° 32.843°1.368 25° 34.187° 1.367 26° 35.529° 1.367 27° 36.868° 1.365 28° 38.203°1.364 29° 39.535° 1.363 30° 40.863° 1.362 31° 42.188° 1.361 32° 43.509°1.360 33° 44.826° 1.358 34° 46.140° 1.357 35° 47.448° 1.356 36° 48.753°1.354 37° 50.053° 1.353 38° 51.348° 1.351 39° 52.639° 1.350 40° 53.924°1.348 41° 55.205° 1.346 42° 56.480° 1.345 43° 57.749° 1.343 44° 59.013°1.341 45° 60.271° 1.339 46° 61.523° 1.337 47° 62.769° 1.336 48° 64.009°1.334 49° 65.242° 1.331 50° 66.468° 1.329 51° 67.687° 1.327 52° 68.899°1.325 53° 70.103° 1.323 54° 71.300° 1.320 55° 72.489° 1.318 56° 73.670°1.316 57° 74.843° 1.313 58° 76.007° 1.310 59° 77.162° 1.308 60° 78.308°1.305 61° 79.445° 1.302 62° 80.572° 1.300 63° 81.689° 1.297 64° 82.796°1.294 65° 83.892° 1.291 66° 84.977° 1.288 67° 86.051° 1.284 68° 87.113°1.281 69° 88.163° 1.278 70° 89.200° 1.274 71° 90.225° 1.271 72° 91.236°1.267 73° 92.233° 1.263 74° 93.216° 1.260 75° 94.185° 1.256 76° 95.137°1.252 77° 96.074° 1.248 78° 96.994° 1.244 79° 97.897° 1.239 80° 98.782°1.235 81° 99.648° 1.230 82° 100.495° 1.226 83° 101.322° 1.221 84°102.127° 1.216 85° 102.910° 1.211 86° 103.670° 1.205 87° 104.405° 1.20088° 105.115° 1.194 89° 105.799° 1.189 90° 106.454° 1.183 91° 107.080°1.177 92° 107.674° 1.170 93° 108.236° 1.164 94° 108.762° 1.157

The mapping functions based on this table to and from visualeccentricities and retinal eccentricities will be called VisEccToRetEcc[] and RetEccToVisEcc[ ]. They can be assumed to be piecewise linearinterpolation between entries of the table above.

This retinal magnification has some important implications.

Generally it means that you can't convert from retinal eccentricities tovisual eccentricities without using the mapping function. But you canconvert visual longitude to retinal longitude, because they are thesame:RetinalLongitude=VisualLongitude  (145)which means:VisualCoordinatesToRetinalCoordinates·Φ[ϕ,θ]=zr·Φ[ϕ,θ]=ϕ  (146)RetinalCoordinatesToVisualCoordinates·ϕ[Φ,Θ]=rz·ϕ[Φ,Θ]=Φ  (147)

where we have used upper case angles for the VisualCoordinates to avoidconfusion.

When converting purely longitudinal angles, the retinal angle and visualangle are the same.

Assuming the retinal radius is the default 12 mm, we can convert theretinal distance between any two points on the surface of the retina (asmeasured along the great circle connecting the two points) into aretinal angle:

$\begin{matrix}{{RetinalAngle} = \frac{RetinalDistance}{12\mspace{14mu}{mm}}} & (148)\end{matrix}$

The retinal angle is in units of radians. But it is handy to know howmany degrees in retinal and visual arc in longitude are equivalent to 1mm:

$\begin{matrix}{{{RetinalAngle}\left\lbrack {1\mspace{14mu}{mm}} \right\rbrack} = {\frac{\left( {180{^\circ}\text{/}\pi} \right) \cdot {mm}}{12\mspace{14mu}{mm}} \approx {{4/77}{^\circ}}}} & (149)\end{matrix}$

It is also handy to know how many millimeters on the surface of theretina are equivalent to one degree of retinal arc:

$\begin{matrix}{{{RetinalDistance}\left\lbrack {1{^\circ}\;{ofretinalarc}} \right\rbrack} = {{12\mspace{14mu}{{mm} \cdot 1}{{^\circ} \cdot \frac{\pi}{180{^\circ}}}} \approx {0.209\mspace{14mu}{mm}}}} & (150)\end{matrix}$

When the retinal angle and the retinal distance are only in thelongitudinal direction, these results also apply to visual longitude:

$\begin{matrix}{\mspace{76mu}{{VisualLongitudinalAngle} = \frac{RetinalLongitudinalDistance}{12\mspace{14mu}{mm}}}} & (151) \\{\mspace{76mu}{{{VisualLongitudinalAngle}\left\lbrack {1\mspace{14mu}{mm}} \right\rbrack} = {\frac{\left( {180{^\circ}\text{/}\pi} \right) \cdot {mm}}{12\mspace{14mu}{mm}} \approx {4.77{^\circ}}}}} & (152) \\{{{RetinalDistance}\left\lbrack {1{^\circ}\;{ofvisuallongitudinalarc}} \right\rbrack} = {{12\mspace{14mu}{{mm} \cdot 1}{{^\circ} \cdot \frac{\pi}{180{^\circ}}}} \approx {0.209\mspace{14mu}{mm}}}} & (153)\end{matrix}$

These also would be the conversions for visual eccentricity, if it werenot for the optical distortion.

In the eccentricity direction, we have to use the table:RetinalEccentricity=VisEccToRetEcc[VisualEccentricity]  (154)VisualEccentricity=RetEccToVisEcc[RetinalEccentricity]  (155)which means:

$\begin{matrix}\begin{matrix}{\begin{matrix}{VisualCoordinatesTo} \\{{RetinalCoordinates}.{\Theta\left\lbrack {\phi,\theta} \right\rbrack}}\end{matrix} = {{zr}.{\Theta\left\lbrack {\phi,\theta} \right\rbrack}}} \\{= {{VisEccToRetEcc}\lbrack\theta\rbrack}}\end{matrix} & (156) \\\begin{matrix}{\begin{matrix}{RetinalCoordinatesTo} \\{{VisualCoordinates}.{\theta\left\lbrack {\Phi,\Theta} \right\rbrack}}\end{matrix} = {{rz}.{\theta\left\lbrack {\Phi,\Theta} \right\rbrack}}} \\{= {{RetEccToVisEcc}\lbrack\Theta\rbrack}}\end{matrix} & (157)\end{matrix}$

where we have used upper case angles for the RetinalCoordinates to avoidconfusion.

For angles less than 30°, VisEccToRetEcc[θ]≈1.38·θ:zr·Θ[ϕ,θ]=VisEccToRetEcc[θ]≈1.38·θ0≤θ<30°  (158)and thus also:

$\begin{matrix}{{{rz}.{\theta\left\lbrack {\Phi,\Theta} \right\rbrack}} = {{{RetEccToVisEcc}\lbrack\Theta\rbrack} \approx {\frac{\Theta}{1.38}\mspace{14mu} 0} \leq \Theta < {30{{^\circ} \cdot 1.38}}}} & (159)\end{matrix}$Now we have:

$\begin{matrix}\begin{matrix}{{{RetinalDistance}\left\lbrack {1{^\circ}\;{ofvisualeccentricity}} \right\rbrack} = {{1.38 \cdot 12}\mspace{14mu}{{mm} \cdot 1}{{^\circ} \cdot \frac{\pi}{180{^\circ}}}}} \\{\approx {0.289\mspace{14mu}{mm}}}\end{matrix} & (160)\end{matrix}$

So we see the effect of the squashing in the visual eccentricitydirection. For visual eccentricities less than 30°: 1° of visuallongitudinal corresponds to 0.209 mm on the surface of the retina, while1° of visual eccentricity corresponds to 0.289 mm on the surface of theretina. And all this is assuming a 12 mm retinal radius. It can be seenwhy various authors, assuming different retinal radii, and perhapstrying to use a single “average” conversion constant for converting fromvisual angles to retinal angles, have used 0.200, 0.250, 0.291, and0.300 mm all for supposedly the same conversion factor. (And differentwide field schematic eyes will ray trace out slightly differentconversion tables.)

Generally, all this means that you can't convert a visual angle to aretinal angle, unless you know the visual eccentricity of the locationwhere the visual angle is measured, and in what direction the visualangle is, and that the visual angle is relatively small in extent.

What the retinal magnification means for a square region on the retina,such as a bounding box around a circularly symmetric cone, is that itwill become squashed in visual space in the visual eccentricitydirection, e.g. a rectangle that is (for low visual eccentricities)about 0.72 times less tall than wide. This mean, in principle, thatthere could be 1.38 times more resolution in the visual eccentricitydirection than in the visual longitudinal direction. We will considerthis point in more detail later.

It also means that if one has a function of retinal eccentricity, as aresome we are about to develop, re-parameterizing the function to be basedon visual eccentricity isn't simple.

Summary: Equations (146) and (156) have defined the conversionVisualCoordinatesToRetinalCoordinates, and equations (147) and (157)have defined the inverse of that conversion:RetinalCoordinatesToVisualCoordinates. An approximate linear mappingreasonable for visual eccentricities less than 30° was defined for theeccentricity components of these conversions in equations (158) and(159).

From these, mapping to and from various other spaces of interest can beconstructed by compositing the appropriate existing mappings.

Cones are not Always “Flat-On” to the Local Surface of the Retina

This subsection does not modify the mapping between the RetinalSurfaceand VisualCoordinates, but because the results show that the aspectratio of cones on the surface of the retina is not unity, it will affectthe mappings of cone cells and midget bipolar cells to theRetinalSurface, and thus also to VisualCoordinates.

To gather the maximum amount of light, cones actively point (e.g.,biological motors actively steer them over the course of a day) in thedirection of the center of the exit pupil of the eye as it appears fromtheir individual location on the surface of the retina. At anyappreciable retinal eccentricity, this will be a somewhat differentdirection than the local normal to the surface of the retina is at thelocation of the cones. Because of this, at higher eccentricities thecones are slightly flattened relative to the local surface of theretina. This has the effect of slightly squashing the visual height ofcones, but this only partially counters the effect of the retinalmagnification, and only at relatively large eccentricities. This doesn'tchange general conversions of visual eccentricity to retinaleccentricity, or vice versa, but it does change the properties of theassumed hexagonal pitch of cone centers. The amount of this squashing isgiven by the cosine of the angle between the ray from a given cone tothe center of the retinal sphere and the ray from the cone to the centerof the exit pupil of the eye. There exist approximate closed formequations describing the amount of local tilt parameterized by (visualor retinal) eccentricity, but ray traced based tables give betterestimates.

The Retinal Size of Cones is Governed by Several Functions

The purpose of this sub-section is to define a function for a circleequitant diameter of retinal cone cells as a function of visualeccentricity. This function will be used in the next sub-section todefine a portion of the RetinalCones manifold mappings to and fromVisualCoordinates.

As previously described, the retinal size of cones is (almost) only afunction of eccentricity. The specific “retinal size” of cones we willcompute is their equivalent circle's diameter in millimeters on theretinal surface. (The equations we have developed can convert thismeasure of “size” to measures of the underlying hexagonal tiling: theshort diagonal and the short pitch.) The function for ConeDiam[θ°] canbe broken up into three different functions over three different rangesof visual eccentricity. The argument θ° is visual eccentricity, in unitsof degrees. (The full name of the function really isRetinalConeDiameterinMM[ ], but we will keep it short.)

Inside the foveal maximum cone density zone, e.g. within two minutes ofvisual eccentricity, the size of cones is determined by the maximum conedensity of the particular individual, which can range from 125,000cones/mm² to 350,000 cones/mm². Since no rods appear between cones here,the cone size and the cone spacing is the same, and can be computed fromthe rule relating the density of hexagons to their equivalent circlediameter (in units of mm on the surface of the retina):

$\begin{matrix}{{{ConeDiam}\lbrack{\theta{^\circ}}\rbrack} = {{\sqrt{\frac{4}{\pi \cdot {FovealMaximumConeDensity}}}\mspace{14mu} 0{^\circ}} \leq {\theta{^\circ}} < {1{^\circ}\text{/}30}}} & (161)\end{matrix}$

Between two minutes and 1° of visual eccentricity, the density of conesdrops from that of the individual foveal maximum cone density to about50,000 cones/mm², regardless of individual variation.

$\begin{matrix}{{{ConeDiam}\lbrack{\theta{^\circ}}\rbrack} = {{\sqrt{\frac{4}{\pi \cdot {{ConeDensity}\lbrack{\theta{^\circ}}\rbrack}}}1{^\circ}\text{/}30} \leq {\theta{^\circ}} < {1{^\circ}}}} & (162)\end{matrix}$Where:

$\begin{matrix}{{{ConeDensity}\lbrack{\theta{^\circ}}\rbrack} = {{{FovealMaximumConeDensity} \cdot \left( {1 - \frac{{\theta{^\circ}} - {1{{^\circ}/30}}}{{1{^\circ}} - {1{{^\circ}/30}}}} \right)} + {50\text{,}{000 \cdot \left( \frac{{\theta{^\circ}} - {1{{^\circ}/30}}}{{1{^\circ}} - {1{{^\circ}/30}}} \right)}}}} & (163)\end{matrix}$

This purely linear fall-off is an approximation, in part because whileno rods appear between cones for visual eccentricities below 0.7°, thisequation applies both to this limited range and beyond out to 1°.

From 1° of visual eccentricity all the way out to the ora serrata, thediameter of cones was modeled by [Tyler, C. 1997. Analysis of HumanReceptor Density, in Basic and Clinical Applications of Vision Science,Ed. V. Kluwer Academic Publishers, 63-71] as:ConeDiam[θ°]=(0.005 mm/°)·(0.2°+θ°)^(1/3)1°≤θ°<ora serrata  (164)

Note again that while the results of ConeDiam[θ° ] is defined in theretinal space, as distance in mm on the RetinalSurface, the parameter tothe function is visual eccentricity, not retinal eccentricity, and is inunits of degrees, not radians. This is the form in which we willeventually want this component of the mapping function in, so we won'tshow the re-parameterization as a function of retinal eccentricity.

The Spacing Between Cones is Governed by Two Functions

The purpose of this sub-section is define the directional magnitudederivative of the RetinalCones manifold mapping to ViewSphereVS.

As previously described, the spacing between cones is (almost) only afunction of eccentricity.

Inside the sub portion of the fovea with less than 0.7° visualeccentricity, the foveal rod-free zone, there are no rods, only cones,and therefore the spacing between cones is determined by the size of thecones. Outside the foveal rod-free zone, rods start appearing betweenthe individual cones, and the spacing between individual cones grows bya different function, which is greater than that determined just by thesize of the cones.

In both cases, the local spacing between cones is a function of thelocal density of cones. Here we assume that the cones keep a relativehexagonal tiling of their centers, even though the cones themselvesdon't directly abut once enough rods start interspersing between them.This means that we can use our previously developed equation forconverting from density and the short pitch of the underlying hexagonaltiling. Because for now we are defining resolution by the Nyquist limit,the uppermost resolution of a hexagonal tiling is determined by theShortPitch, although this resolution will only occur in the localdirection of the ShortPitch within the local tiling. Restating therelationship between the density of hexagons and the length of theShortPitch:

$\begin{matrix}{{RetinalConeShortPitchInMM} = \sqrt{\frac{\sqrt{3}}{2} \cdot \frac{1}{{RetinalConeDensityPerMM}\; 2}}} & (165)\end{matrix}$

This equation can be used to determine the ShortPitch of cone spacinggiven a cone density function of visual eccentricity in units ofdegrees:

$\begin{matrix}{{{RetinalConeShortPitchInMM}\lbrack{\theta{^\circ}}\rbrack} = \sqrt{\frac{\sqrt{3}}{2} \cdot \frac{1}{{RetinalConeDensityPerMM}\;{2\lbrack{\theta{^\circ}}\rbrack}}}} & (166)\end{matrix}$

Where the density is measured in units of cones/mm², so the ShortPitchwill be in units of mm of length, thus the long names.

Now we need to define the cone density function. Between 0° and 1° ofvisual eccentricity, the density of cones will be assumed to be the sameas was assumed when determining the size of cones above, so we have:

$\begin{matrix}{{{{RetinalConeDensityPerMM}\;{2\lbrack{\theta{^\circ}}\rbrack}} = {FovealMaximumConeDensity}}\mspace{20mu}{{0{^\circ}} \leq {\theta{^\circ}} < {1{{^\circ}/30}}}} & (167) \\{{{{RetinalConeDensityPerMM}\;{2\lbrack{\theta{^\circ}}\rbrack}} = {{{FovealMaximumConeDensity} \cdot \left( {1 - \frac{{\theta{^\circ}} - {1{{^\circ}/30}}}{{1{^\circ}} - {1{{^\circ}/30}}}} \right)} + {50\text{,}{000 \cdot \left( \frac{{\theta{^\circ}} - {1{{^\circ}/30}}}{{1{^\circ}} - {1{{^\circ}/30}}} \right)}}}}\mspace{20mu}{{1{{^\circ}/30}} \leq {\theta{^\circ}} < {1{^\circ}}}} & (168)\end{matrix}$

Between 1° and 20° of visual eccentricity, we will use the [Tylor 1997]model of cone density:RetinalConeDensityPerMM2[θ°]=50,000.0·θ°^(−2/3)1°≤θ°<20°  (169)

At 20° of visual eccentricity, the density of cones by the equationabove will have fallen to ^(˜)6,800 cones/mm². From 20° of visualeccentricity, out to 60° of visual eccentricity, we will follow Tyler'ssuggestion that cone density fall off linearly from the previous to^(˜)6,800 cones/mm² to 4,000 cones/mm²:

$\begin{matrix}{{{{RetinalConeDensityPerMM}\;{2\lbrack{\theta{^\circ}}\rbrack}} = {{50\text{,}{000 \cdot 20}{{^\circ}^{{- 2}/3} \cdot \frac{\left( {{60{^\circ}} - {\theta{^\circ}}} \right)}{40{^\circ}}}} + {4\text{,}{000 \cdot \frac{\left( {{\theta{^\circ}} - {20{^\circ}}} \right)}{40{^\circ}}}}}}\mspace{20mu}{{20{^\circ}} \leq {\theta{^\circ}} < {60{^\circ}}}} & (170)\end{matrix}$

At 60° of visual eccentricity, the density of cones by the equationabove will have fallen to 4,000 cones/mm². From 60° of visualeccentricity, out to the edge of the ora serrata, we will keep the conedensity constant at 4,000 cones/mm²:RetinalConeDensityPerMM2[θ°]=4,000 60°≤θ°<ora serrata  (171)

Equations (167), (168), (169), (170), and (171) allow equation (166) todefine the ConeShortPitch in units of mm for the entire range ofpossible visual eccentricities. We would prefer to have theConeShortPitch in units of visual angle, so first we need to convert theoutput from units of mm on the surface of the retina to a retinal anglein units of radians by dividing by the (default) retinal radius:

$\begin{matrix}{{{ConeShortPitchInRetinalAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} = \frac{{RetinalConeShortPitchInMM}\lbrack{\theta{^\circ}}\rbrack}{12\mspace{14mu}{mm}}} & (172)\end{matrix}$

Now we need to convert from the retinal angle to the visual angle. Butthis conversion depends on the direction of the short pitch, as well asthe visual eccentricity.

In the purely longitudinal direction, the retinal angle and the visualangle are the same, so we get:

$\begin{matrix}{{{ConeShortPitchInVisualLongitudinalAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} = \frac{{RetinalConeShortPitchInMM}\lbrack{\theta{^\circ}}\rbrack}{12\mspace{14mu}{mm}}} & (173)\end{matrix}$

In the purely eccentricity direction, the visual angle will be minifiedby:

$\begin{matrix}{{{ConeShortPitchInVisualEccentricityAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} = {\frac{\theta{^\circ}}{{VisEccRetEcc}\lbrack{\theta{^\circ}}\rbrack} \cdot \frac{{RetinalConeShortPitchInMM}\lbrack{\theta{^\circ}}\rbrack}{12\mspace{14mu}{mm}}}} & (174)\end{matrix}$

How do we take a function that gives us the short pitch of a cone at anyeccentricity and use it to construct the RetinalCones manifold? Asdescribed earlier, we start with the directional magnitude derivative.Since we are using our mappings to define resolution, and because thehighest resolution that a hexagonal array of cones can perceivedescribed by their short pitch, we can take the short pitch function ineach angular direction and define them to be the directional magnitudederivatives in those directions:

$\begin{matrix}{{{\nabla_{\phi}{RetinalCones}} \cdot {sv} \cdot {\phi\left\lbrack {\phi,\theta} \right\rbrack}} = {{{ConeShortPitchInVisualLongitudinalAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} = \frac{{RetinalConeShortPitchInMM}\lbrack{\theta{^\circ}}\rbrack}{12\mspace{14mu}{mm}}}} & (175) \\{{{\nabla_{\theta}{RetinalCones}} \cdot {sv} \cdot {\phi\left\lbrack {\phi,\theta} \right\rbrack}} = {{{ConeShortPitchInVisualEccentricityAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} = {\frac{\theta{^\circ}}{{VisEccToRetEcc}\lbrack{\theta{^\circ}}\rbrack} \cdot \frac{{RetinalConeShortPitchInMM}\lbrack{\theta{^\circ}}\rbrack}{12\mspace{14mu}{mm}}}}} & (176)\end{matrix}$

Note that the mapping that we are taking the directional magnitudederivative of is sv, e.g. RetinalCones·surfaceToViewSphereVS. That isbecause a visual angle is an angle on the ViewSphere, which even iftaken in the longitudinal direction should still be based on thedistance between two points on the surface of the ViewSphere as measuredalong the great circle that connect them, not any longitudinal angle inVisualCoordinates.

For our purposes here, these directional magnitude derivatives are theresults we will want for use in the next section. So we don't need tofollow through with the integration of the piecewise defined functions.It is enough to observe that the results of such integration will belinear functions of θ°, linear functions of θ°², and a function ofθ°^(1/3). This all shows what we already know: while the density ofcones goes down with increasing visual eccentricity, e.g. the spacingbetween the cones gets larger, the actual number of cones straddling anyparticular visual eccentricity on the ViewSphere goes up.

We will also note that a full specification of the Cones mapping wouldrequire the creation of an EndCap mapping for the foveal maximum conedensity zone.

The Number of Cones that Contribute to the Center Input of RetinalMidget Ganglion Cells Varies with Eccentricity

The purpose of this sub-section is to define the directional magnitudederivative of the Retinal Midget manifold mapping to ViewSphereVS. Oncewe have this, we can then compare how close to a locally uniformresolution mapping the RetinalMidget mapping is. But first we willobtain an equation for how many cone cells there are, on average,feeding into the center field of midget ganglion cells, as a function ofvisual eccentricity.

Inside a visual eccentricity of 6°, each midget ganglion cell obtainsits center input from precisely one cone, so the underlying tiling ofthe midget ganglion cells is the same as the tiling of the underlyingcones (which we are still assuming to be hexagonal):

$\begin{matrix}{{{{MidgetShortPitchVisualLongitudonalAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} = {{ConeShortPitchVisualLongitudonalAngleInRadians}\lbrack{\theta{^\circ}}\rbrack}}\mspace{20mu}{{0{^\circ}} \leq {\theta{^\circ}} < {6{^\circ}}}} & (177) \\{{{{MidgetShortPitchVisualEccentricityAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} = {{ConeShortPitchVisualEccentricityAngleInRadians}\lbrack{\theta{^\circ}}\rbrack}}\mspace{20mu}{{0{^\circ}} \leq {\theta{^\circ}} < {6{^\circ}}}} & (178)\end{matrix}$

Beyond 6° of visual eccentricity, the center input to midget ganglioncells start to come from more than one cone cell, with the number ofsuch cones increasing more and more with increasing eccentricity.[Dacey, Dennis M. “The Mosaic of Midget Ganglion Cells in the HumanRetina, The Journal of Neuroscience, December 1993, 13(12): 5334-5355]studied the diameters of midget ganglion cells and fit an equation tohis anatomical measurements (from 0.5° to 75° of visual eccentricity).The equation gives the diameter of midget ganglion cells expressed as avisual angle in units of minutes of arc as a function of visualeccentricity measured in units of degrees:

$\begin{matrix}{{{{VisAngDiameterInArcMin}\lbrack{\theta{^\circ}}\rbrack} = {2.1 + {0.058 \cdot {\theta{^\circ}}} - {0.022 \cdot {\theta{^\circ}}^{2}} - {0.00022 \cdot {\theta{^\circ}}^{3}}}}\mspace{20mu}{{0.5{^\circ}} \leq {\theta{^\circ}} < {75{^\circ}}}} & (179)\end{matrix}$

Dacey's empirical data about the location of midget ganglion cells onthe retinal surface was measured in units of retinal distance from thecenter of the fovea in millimeters (rdist). To convert these distancesto visual eccentricities, he used a simplified formula of:θ°=0.1+3.4·rdist°+0.035·rdist°²  (180)

where θ° is visual eccentricity in units of degrees.

Dacey measured the diameter of a midget ganglion cell by empiricallydrawing a polygon around its dendritic field, then calculating thepolygon's area, and calculating the diameter of the circle that wouldhave the same area (just as we have done in the hexagon case). Thisdiameter is a (short) retinal distance measured at a particular retinaleccentricity. How he converted this retinal distance to a visual angleis not explicitly stated. From his plots it looks like he obtained thelocal conversion factor from retinal angle to visual angle using thederivative of equation (180). This is the correct conversion forconverting a retinal distance in the eccentricity direction to a visualangle in the eccentricity direction, and represents the highest possibleresolution. However, it is not the correct conversion in thelongitudinal direction, where the conversion factor is a constant 0.209mm/°. So we will use his diameter only as a measure in the eccentricitydirection.

First we need to convert the output from visual angle in units ofminutes of arc to visual angle in units of radians:

$\begin{matrix}{{{VisAngDiameterInRadians}\lbrack{\theta{^\circ}}\rbrack} = {\frac{\pi}{60 \cdot 180} \cdot \left( {2.1 + {0.058 \cdot {\theta{^\circ}}} + {0.022 \cdot {\theta{^\circ}}^{2}} - {0.00022 \cdot {\theta{^\circ}}^{3}}} \right)}} & (181)\end{matrix}$

We can convert this to the length of the ShortPitch of the underlyinghexagonal tilling via the relationship:

$\begin{matrix}{{MidgetShortPitch} = {{\frac{\sqrt{\pi}}{2} \cdot \sqrt{\frac{\sqrt{3}}{2}} \cdot {diameter}} \approx {0.825 \cdot {diameter}}}} & (182)\end{matrix}$

Therefore (and since this equation is only good in the eccentricitydirection):

$\begin{matrix}{{{MidgetShortPitchVisualEccentricityAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} = {{\frac{\sqrt{\pi}}{2} \cdot \sqrt{\frac{\sqrt{3}}{2}} \cdot \frac{\pi}{60 \cdot 180}}\left( {2.1 + {0.058 \cdot {\theta{^\circ}}} + {0.022 \cdot {\theta{^\circ}}^{2}} - {0.00022 \cdot {\theta{^\circ}}^{3}}} \right)}} & (183)\end{matrix}$

Below a visual eccentricity of 6°, our modified Dacey's equation also is(should be) an equation for the short pitch of the tiling of cones onthe retina. Thus Dacey's fitting of a curve to empirical data aboutmidget ganglion cell density in the 0.5° to 6° range of visualeccentricity should be comparable to Tyler's fitting of Oesterberg'sempirical data about the cone cell density in the same range. Simpleinspection of their equations shows that they can't match exactly,Tyler's is curve to the ⅓^(rd) power of the visual eccentricity; Dacey'sis a cubic polynomial in the visual eccentricity. FIG. 89 shows a plotof the predicted cone short pitch measured in minutes of arc in thevisual eccentricity direction against degrees of visual eccentricity forTyler, the thick line, element 8920, and for Dacey, the thin line,element 8910. Because the two equations predict different absolute coneshort pitches, in this plot they have been normalized to meet at 4°,where the two curves are tangent. Some of this difference is due toconverting between longitudinal direction and eccentricity directioncone diameters. Both curves predict similar straight lines, thoughTyler's seems to be predicting some of the ramp up to foveal densitiesbetter.

With the above caveat, we will now proceed to use the square of theratio of Dacey's equation to that of Tyler's in the range of 4° to 48°to see what they predict about how many cones, on average, feed into asingle midget ganglion cell center input at a given visual eccentricityθ° (in units of degrees). We need take the square of the ratio of midgetganglion cell ShortPitch to ConeShortPitch. Because the two modelsshould not predict more cones than midget ganglion cells at 6°, thistime we normalize the results of each function at that angle:

$\begin{matrix}{{{ConesPerMidgetCenter}\lbrack{\theta{^\circ}}\rbrack} = \begin{pmatrix}{\frac{{MidgetShortPitchVisualEccentricityAngleInRaidans}\lbrack{\theta{^\circ}}\rbrack}{{ConeShortPitchVisualEccentricityAngleInRadians}\lbrack{\theta{^\circ}}\rbrack} \cdot} \\\frac{{ConeShortPitchVisualEccentricityAngleInRadians}\left\lbrack {6{^\circ}} \right\rbrack}{{MidgetShortPItchVisualEccentricityAngleInRadians}\left\lbrack {6{^\circ}} \right\rbrack}\end{pmatrix}^{2}} & (184)\end{matrix}$

In Table 3 below, we show at which visual eccentricities in units ofdegrees does the number of cone cells per single midget ganglion cellreceptor field center achieve successive integer values, given the twomodels.

TABLE 3 Visual eccentricity vs. number of cone inputs to midget ganglioncenter field. Visual Eccentricity Number of Cones 6.0° 1 12.3° 2 15.8° 318.5° 4 20.8° 5 22.9° 6 24.7° 7 26.4° 8 28.1° 9 29.6° 10 31.1° 11 32.6°12 34.0° 13 35.4° 14 36.8° 15 38.2° 16 39.6° 17 41.1° 18 42.5° 19 44.0°20 45.6° 21 47.3° 22

We can compare the predictions of the short pitch by the locally uniformresolution mapping against that of Dacey's, as well as the LogPolarmapping. FIG. 90, thick line, element 9010, shows the ratio of thepredicted short pitch of locally uniform resolution mapping overDacey's. FIG. 90, thin line, element 9020, shows the ratio of thepredicted short pitch of the LogPolar mapping over Dacey's. Both areshown over the visual eccentricities of 6° to 75°. Both mappings havetrouble in the 6° to 10° region. The locally uniform resolution mappingstays within 62% of Dacey over the wide range of 10° to 75°, but the cos[ ] term in the LogPolar mapping to steadily drop to below 20% of Dacey.This again shows that the LogPolar mapping doesn't match very well whatseems to be happening in the human eye (and the visual cortex of thebrain) at larger angles. Again, in this plot, we are comparingpredictions of short pitch length, which is the inverse of resolution.Thus the fact that both mappings fall to lower values than Dacey'sempirical measurements means that both are predicting higher resolutionthan Dacey's data supports. Both mapping models were normalized to unityat the point of highest match to Dacey at 10°. So it is also possiblethat at least for the locally uniform resolution mapping, it might bepredicting lower resolution than Dacey around 10°, and somewhat moreconsistent numbers at greater visual eccentricities. Another factor toconsider is what portion of the visual field in visual longitude are thevarious models based on. Tyler considers all longitudes, but his modelseems to be focused on the nasal quadrant. Dacey takes data from thefour major longitudes separately, but his curve fit is to the data fromall longitudes. The locally uniform resolution mapping is fit to thenasal quadrant.

Now to put all of the data into the form of resolution curves, whichwill be the inverse of the short pitch prediction curves of the lastparagraph. FIG. 91 shows resolution, in units of cycles per degree, onthe vertical axis, verses visual eccentricity, in units of degrees, onthe horizontal axis. The filled black circles, element 9110, aremeasurements of human visual resolution from [Anderson S J, Mullen K T,Hess R F (1991) Human peripheral spatial resolution for achromatic andchromatic stimuli: limits imposed by optical and retinal factors. JPhvsiol (Land) 442:47-64.]. The curved thick line, element 9130, is theresolution of the locally uniform resolution mapping with a SW of 720equivalent square pixels of resolution. The curved thin line, element9120, is the resolution of Dacey's midget ganglion cell model. Forreference, we include the resolution predicted by the retinal conemosaic as the curved dashed line, element 9140, based on the cones shortpitch from Tyler's model.

In his paper Dacey showed that his model of midget ganglion cell visualangle in “close” approximation with experimental measurements of humanperceptual resolution. This can be seen in FIG. 91 in how close hiscurve 9120 follows all but the 5° lowest visual eccentricity data point.The locally uniform resolution mapping 9130 is also a close fit, exceptat the same 5° data point. The curve for the cone predicted resolution9140 overshoots the human experimental perception except at the 5° datapoint. This is as expected, as beyond 6°, human perceptual resolution isdriven by the resolution of the midget ganglion cells, not the conecells. To avoid over cluttering, the LogPolar mapping's resolutionprediction was not included in this figure. For the range of visualeccentricities shown, e.g. below 35°, the LogPolar mapping overshootsthe human curve only be a little. As was shown in FIG. 90, at highervisual eccentricities, the cos [ ] term causes the LogPolar mapping tomuch further overshoot.

This last figure shows the locally uniform resolution mapping in itsbest light. There was only one parameter to fit, and that was SW. Thisparameter really just sets the midget ganglion cell density at onepoint.

The fact that the models over-estimate resolution below 10° of visualeccentricity will be addressed in the next section.

Comments on Estimating Perceptual Resolution Based on Receptor ShortPitch

In the visual science literature it is common to find remarks about howour highest visual acuity at the center of the fovea occurs at the pointwhere the three major factors limiting resolution all meet: the limitsimposed by classical optical quality of the eye's optical system, thelimits imposed by diffraction at the smallest pupil size, and the limitsimposed by the retinal sampling mosaic (e.g., the hexagonally tiledcones). Sometimes a fourth matching limit is added: that of neural noise(though there are several different interpretations of this). All thisis quite important, as the mapping implied by the resolution of the eyenot only (is thought to) determines the density of the retinal midgetganglion cells, but also the physical organization and structure of mostof the visual cortex (one third of the entire human brain). However,there is ample evidence that this triple match point does not actuallyoccur, with the implication that human visual resolution is not (always)determined simply by the density of the retinal midget ganglion cells.This sub-section will describe why the two are somewhat decoupled,especially near the fovea.

In the fields of image processing and computer vision, a big changeoccurred many years ago when vacuum tube based video cameras(essentially the inverse of a CRT) started to be replaced by discretedigital pixel semiconductor based sensors. These early digital sensorshad relatively large pixels with somewhat low fill-factors (gaps betweenthe active area of one pixel to the next), and it was rapidly discoveredthat processing worked better of the image was slightly defocused. Thiswas usually done optically, but could also be accomplished by an analoglow pass filter on the analog video signal (which the early digitalcameras still generated), or later, as a digital low pass filter on thedigital pixel data. Even with today's much higher resolution digitalcamera chips, this is still true. Why?

The Nyquist theorem about the maximum frequency component that can beextracted from a (one dimensional) signal comes with a substantialcaveat: that signal component must repeat at the same frequencyinfinitely to the left and right of the sample point. In practice,infinity is not required, but a convolution window of 17 or more pixelsis. Why? Consider a one dimensional sine wave whose period exactlymatches the spacing of two samples (e.g., at the Nyquist limit).Assuming that the peak of the sine wave occurs centered over all theeven sample intervals (a “sample interval” is a one dimensional pixel),and that the trough occurs centered over all the odd sample intervals.What is the digitally sampled result? There sequential digital sampleswill contain high valued pixels followed by low valued pixels; thesignal has successfully been captured (thought at a slight reduction incontrast). However, consider what happens if the sine wave is shifted inphase by half a sample interval. Now all sample intervals will see halfof a peak and half of a trough, or vice versa, but all samples will endup sampling the same value (the average of the sine wave). The resultantdigital sample stream will be constant, absolutely no evidence of thesine wave will be present in the digital domain. At phase offsets otherthan exactly half a sample interval, evidence of the underlying sinewave will return, but at reduced to greatly reduced contrast.Practically, a sine wave just a little lower than the Nyquist limit,sampled over 17 sample intervals, and re-constructed over the samerange, will bring back most of the signal. But in visual space, one islucky to get a quarter of a period of a frequency: a rounded step edgebetween a dark region and a light region. Less frequently, one mayencounter half of a period of frequency, e.g., a line: a step upfollowed by a step down. Thus in practice, the Nyquist limit is too highof bar to set on what peak resolution can be expected to be reliably(e.g., not aliased) extracted from a given sample interval. This is whypeople defocused the lenses on their digital cameras, to pre-limit themaximum spatial frequency of the image impinging on the digital sensorto something below the Nyquist limit. How far below? At 30% to 50% belowthe Nyquist limit even step edges now no longer disappear, and themaximum reduction in their contrast is reasonable (e.g. less than 50% orso). A related reason to reduce the maximum spatial frequency present isthat the Nyquist theory applies to one dimensional signals, there is nota simple 2D extension. The closest would be to look at sine wavepatterns extended in space, presented at all possible angles relative tothe underlying tiling of the sensing pixels, and choose which has thelowest resolution. Another way of saying this is that the use of theshort pitch of a tiling as the sample interval was a “best case” result.At other orientations, the sample interval is lengthened to the longpitch (the short diagonal), and it can be argued that in the worst casethe sample interval could be reduced to the long diagonal of tiling.Considering both factors: the requirement to see reasonable contrast ofstep edges at all phases, and the lengthening of the sample interval atmost orientations, reducing the maximum expected spatial frequency thatcan be expected to be reliably extracted from a pixel tiling by even asmuch as 30% over the Nyquist rate implied by the short pitch of thetiling seems a bit optimistic.

There is ample evidence that the human eye does impose such limits byoptical means in the foveal region. While aliasing effects have beendetected in the periphery, they have never been detected in the foveaunder natural viewing conditions. The proof that the retinal image hashad the maximum spatial frequency ever present low-passed by the opticsof the eye come from experiments that effectively by-pass the eye'snatural optical limits. When real-time deformable mirrors are used tocorrect for the optical imperfection of the human eye with anartificially dilated large pupil (to avoid the limits imposed bydiffraction), aliasing has been observed in the fovea. This means thatthe low-pass filtering is not performed at the neural level of coneoutputs (though some charge sharing does occur). Rather, the optics ofthe human eye appear to be intentionally kept at a slightly lowerquality. (Birds, with magnified foveas, have considerably higher qualityoptical elements than humans do, even though both are formed out ofhighly modified skin cells.) The amount of low-pass filtering caused bythe human eye's optics in the foveal region can be quantified by variousexperiments, and has been found to be in the 30%+ reduction range overthe short pitch Nyquist limit.

What this means for models of the resolution of the human eye is that itshould not be expected that human perception resolution in the fovealregion should match what the Nyquist limit implied by the short-pitch ofthe underlying cone tiling would directly predict (and just in thepre-shortened retinal eccentricity direction). The underlyingorganization of the retinal midget ganglion cells and thus also theimplied spatial organization of the visual cortex in the foveal regioncan still be correctly predicted by the cone tiling; it is only theperceptual resolution in the foveal region that will instead bepredicted by a model of the optics of the eye in that region.

What about the resolution in the periphery? Outside the foveal region,the quality of the eye's optics drops relatively fast, but not as fastas the density of the receptor tiling implied by the midget ganglioncells drops. This implies two things: one, aliasing effects should occurin the periphery (they do), and two, the resolution in the peripheryshould be more directly tied to the Nyquist limit determined by theshort pitch of the midget ganglion cells (it is).

What does all this mean for the locally uniform resolution mapping?Because of its local preservation of shape, it still is a good mappingfor man-made (display and image processing) applications to use outsideof any EndCap, and outside the large foveal region (˜8°+ of visualeccentricity), it is still a good candidate for modeling what the humaneye's midget ganglion cell mapping is. The “failure” of the locallyuniform resolution mapping near the foveal region is thus a non-issue,as it is caused by other factors. Below 6° of visual eccentricity, theretinal density of midget ganglion cells is constrained to be the sameas the underlying cone density, as in this region each midget ganglioncell connects to exactly one cone. Starting above 6° of visualeccentricity, midget ganglion cells can attach to multiple(statistically including an integer+fractional) numbers of cone cells,and the density of midget ganglion cells can follow whatever curveevolution wants, relatively independent of the cone density. It makessense then that the mapping of midget ganglion cells density over alleccentricities is not a single simple function of eccentricity, but atleast two, with the locally uniform resolution mapping a good candidateabove 8°. (Why 8°, not 6° ? The two mapping functions have to meet at6°, and the curve is modified in this region.)

Comments on the Biological Plausibility of the Locally UniformResolution Mapping

The locally uniform resolution mapping may have optimal properties thatwould make it a good choice for use by biological systems for mappingthe midget ganglion cells, but is there a simple way that such systemscould “implement” it? It turns out that just setting a biologicalconstraint that the number of midget ganglion cells at placed around alllongitudes any retinal eccentricity be constant, and that the cellspacing in the eccentricity direction be the same as in the longitudinaldirection, is enough to automatically form a locally uniform resolutionmapping. There are also some evolutionary arguments that can be made,but won't be gone into here.

Comments on the Non-Longitudinally Symmetric Resolution of the Human Eye

The locally uniform resolution mapping as presented so far has alwaysbeen a LinearlyLongitudinal mapping. This is the preferred structure forphysical implementations of contact lens displays that must berotationally symmetric, and is also appropriate for image processing andcomputer vision applications. It also is integral pixel structurepreserving. But it is known that the human eye and human perceptualresolution is not a LinearlyLongitudinal mapping: we have higherresolution in the nasal direction. Full details on how the locallyuniform resolution mapping can be extended to be notLinearlyLongitudinal, and can be fit to human perception data in allquadrants, will not be given here. Instead it will just be pointed outthat human perceptual resolution (and the underlying density of midgetganglion cells) appears to be close to a locally uniform resolutionmapping along at any fixed longitude. It is just that the parameter ofthe mapping (SW) slowly varies at different longitudes; it is higher inthe nasal direction, less so in the temporal, and even less in thesuperior and oblique directions. So a sketch of how theLinearlyLongitudinal mapping could track this would be one in whichpixel columns at different values of u (longitude) would map to smalleramounts of incremental longitudinal angle (outside the symmetric fovealEndCap mapping). This still (mostly) preserves the pixel structure, willallowing the resolution to no longer by constant at the same visualeccentricity at all visual longitudes.

V. Optical Function Required for Eye Mounted Displays

Where an eye mounted display is placed: outside, on top of, or withinthe eye, determines the required optical function of such a display. Asshown in FIG. 9 and FIG. 14, in the physical world, and in nearly allprevious display technologies, point sources of light 940 and 1410produce expanding spherical waves of light 950 and 1420. Some of thesewavefronts will eventually intersect with the cornea of a humanobserver's 920 eye. At that point the wavefront will have a radius ofcurvature equal to the distance between the point source and theviewer's cornea. For very distant point sources, e.g. more than a dozenmeters away, the wavefronts are quite flat, and can be treated as planewaves. For very close point sources, e.g. only a foot away, thewavefront is highly curved. Indeed, without external aid, except in highmyopias, the human eye cannot focus on objects much closer than a footaway. In summary, in the natural world, wavefronts of light encounteringthe human eye are expanding spherical wavefronts with a radius at theeye of between one foot and infinity.

Most existing display technologies mimic the natural world: they producesimilar expanding wavefronts of light. Simplistically, one can imagineeach discrete pixel of a display as a different point source. An exampleof this is shown in FIG. 10, where a human element 1020 is viewing twoflat panel displays element 1010, and a pixel on one of the displayscreates a point source element 1030, which generates sphericalwavefronts of light element 1040. More accurately, each pixel isactually a two dimensional region within which there are many differentpotential point sources, each with their own unique sub-pixel location.Projection displays are not viewed directly, instead light from them isfocused onto some form of screen, either front or rear, and pointsources are generated by the screen surface. Head mounted displaysdiffer slightly, in that very small radius wavefront of light from theirinternal pixels are optically flattened by a large amount before theyemerge from the display. This is shown in FIG. 11, where a human element1120 is wearing a head mounted display element 1110. On a small internalflat panel display element 1130, a pixel produces a point source oflight element 1140. This produces small radius spherical wavefronts oflight element 1150. But before reaching the viewer's eye, an opticalelement 1160 within the head mounted display greatly enlarges the radiusof curvature of these spherical wavefronts to produce new sphericalwavefronts element 1170. These “flattened” wavefronts (now effectivelythe same as element 1040) are the ones that reach the eye, where theyare perceived as object much further away than the flat panel element1130. The purpose of this flattening is just to mimic the natural world:to ensure that the wavefronts emitted by the display have a radius ofcurvature somewhere in the human usable range: one foot to infinity.FIG. 12 is a zoom-in of detail of FIG. 11.

An eye mounted display that has an air gap between the display and thecornea is subject to the same laws of physics, and must produce the sameradius wavefronts as described above. But an eye mounted displayattached to the cornea, or placed further within the eye itself, facesdifferent optical constraints.

In the normal un-aided eye, there are just two major optical elements.As shown in FIG. 13 in black, these are the cornea 1310 and the lens1320. We will next step through how these two elements affect theincoming expanding spherical wavefronts of light from the environmentencounter first the cornea 1310 and then the lens 1320.

FIG. 14 shows a point source of light (element 1410) emitting expandingspherical wavefronts of light (element 1420) eventually impacting on thefront surface of the cornea (element 1310). (For aid in illustration,the point source as depicted is much closer to the human eye than it cannormally focus.) As shown in FIG. 15, when these expanding wavefronts oflight from outside the eye pass through the cornea 1310, the opticalfunction of the cornea 1310 is to convert these wavefronts 1420 intopost-corneal contracting spherical wavefronts of light 1510. The radiusof curvature of these contracting waves is small: slightly more than aninch. This means that without further interference they would contractto a single point slightly behind the eye. The cornea has abouttwo-thirds of the optical power of the eye. The last third is containedin the lens 1320. In FIG. 16 it is shown that when the post-cornealwavefronts 1510 pass through the lens 1320, the optical function of thelens 1320 is to convert these wavefronts 1510 into post-lens contractingspherical wavefronts of light 1610. These wavefronts will have a furtherreduction in the radius of curvature over those of the post-cornealwavefronts 1510 such that the post-lens wavefronts 1610 will nowcontract to a point at the surface of the eye's retina (when in focus).This point is shown in FIG. 17 as element 1710.

To a first approximation, when a contact lens is placed on the eye, iteliminates the cornea as an optical element, replacing that with its ownoptical properties. This is because the index of refraction of thecontact lens, tear fluid, and the cornea are very similar, and so littlebending of light is caused by a contact lens covered cornea. Thecornea's natural optical function is caused by the large difference inindex of refraction between air and the material of the cornea. When acontact lens is present, the large change in optical index is no longerat the surface of the cornea, but at the air-contact lens interface.This is illustrated in FIG. 18, which shows in black the two majoroptical elements of a contact lens aided eye: the contact lens 1810 andthe lens 1320. For a myopic or hyperopia, this allows a properlydesigned contact lens to correct one's vision. But there are alsoconsequences for a contact lens display: if a display embedded within acontact lens is emitting wavefronts of light, those wavefronts of lightwill not be modified by the cornea, even though they still pass throughit. Instead, the emitted wavefronts of light must match the naturalpost-corneal wavefronts. As described above, these are rapidlycontracting spherical wavefronts of light, and this is what the opticalsystem of a contact lens display must produce. This is illustrated inFIG. 19, where a single femto projector 1910 is shown embedded withinthe contact lens 1810, and is generating post-corneal wavefronts oflight 1510. FIG. 20 is a zoom into FIG. 19, showing the optics in moredetail.

Besides residing in an external contact lens, eye mounted displays ingeneral could be placed at many other locations within the eye. Eachdifferent placement potentially has a different target type of lightwavefront to produce, and thus different optical function requirements.The general rules is the same: produce the same sort of wavefronts thatthe natural world produces at a given location within the eye's opticalsystem. An eye mounted display placed within the cornea, replacing thecornea, or placed on the posterior of the cornea, all must producesimilar optical wavefronts as a contact lens display. Intraoculardisplays, placed somewhere between the cornea and the lens, have toproduce even tighter radius contracting wavefronts of light, the exactradius depends on the exact location of the intraocular display betweenthe cornea and lens. Eye mounted displays placed on the front of thelens, within the lens, replacing the lens, or on the posterior of thelens, have yet different optical function requirements, which can differbetween the types. Eye mounted displays placed between the lens and theretina need to produce still tighter radius of curvature for properfunction. Eye mounted displays placed on the surface of the retina needto mimic the focused point of light of natural vision, and thusoptically can be just point sources. That is, direct emission of lightby pixels can be sufficient, without any additional opticalmodifications.

With the exception of an air-gap eye mounted display, whose requiredoptical function is otherwise well understood, and retinal mounteddisplays, who may not require any additional optical function at all;all the other cases of possible eye mounted display share the samegeneral requirement: produce contracting small radius sphericalwavefronts of light. They only differ in what small radius they mustgenerate. In the general case in which support is to be included is toproduce different depths of field, and/or programmable opticalprescriptions, rather than a fixed single radius, instead a small rangeof radii to be produced is the target. Thus without loss of generality,we will expand on the optical implementation of a contact lens display,as the other cases require only slight well understood modifications.

VI. Optical Function of a Contact Lens Display

While there are many different possible way of designing a contact lensdisplay, one method would be to use a planer array of light emittingpixels (point or very small area sources of light), followed by anoptical system that converts these very small radius expanding sphericalwavefronts of light (because all this is happening within the smallconfines of a contact lens) into the desired post-corneal contractingwavefronts.

Placed in front of a light emitting pixel array, a single simple convexlens will convert the expanding spherical wavefronts from each pixelinto contracting spherical wavefronts of light. Unfortunately the radiusof the contracting wavefronts will be similar to the incoming expandingwavefronts, and thus will poses a far smaller radius of curvature thanthe required post-corneal natural wavefronts. But by placing a simpleconcave lens just after the convex lens, the resultant radius ofcurvature is greatly expanded, and the desired wavefronts can beproduced. Different numbers and combinations of lenses can produce thesame result. These optical systems may include, for example, GRINlenses, mirror elements, prism elements, Fresnel elements, diffractiveelements, holographic elements, negative index materials, etc.

There are several constraints on the physical size and placement of theoptics within contact lens display; most of these constraints also applyto other types of eye mounted displays. Physically a comfortable contactlens must be fairly thin—many commercial contact lenses are on the orderof 150 microns (0.15 millimeters) thick. Sclera contact lens can bethicker and still comfortable, but the limit is about 500 microns (halfa millimeter). The optical zone of a contact lens is only about 8millimeters in diameter (technically this is a function of the absolutesize of the eyeball). This means that any portion of a contact lensdisplay that is emitting light meant to pass into the eye must belocated within the 8 millimeter diameter circle. However, other portionsof a contact lens display can reside within areas of the contact lensoutside this circle.

In traditional optical terminology, the human eye is a very wide fieldof view device; as much as 165° across. This means that at anyparticular point on the surface of the cornea, light emitted at thatpoint cannot reach the entire retina, but only a specific portion of it.At the center of the cornea this portion can be as wide as 60° across,but at points with higher eccentricity the addressable portion of theretina narrows considerably. This same optical property means that if acontact lens display is meant to be see-through, the size and shape ofany opaque region within the optical zone must be small.

The inability of a single point, or small region, to illuminate most ofthe retina implies that any wide field of view contact lens display mustemit light into the eye from several different locations within theoptical zone. The see-through requirement implies that each such lightemitting component must be fairly small in size: generally less thanhalf a millimeter across. This last constraint precludes many foldedoptics designs. As shown in FIG. 75, a simple in-line optical design,consisting of the display element (light emitting pixels) 7510, a first(positive) lens 7520, and then a second (negative) lens 7530, has theproperty that when the diameter of the system is constrained to be lessthan half a millimeter across, it is possible to design the opticalsystem such that its thickness is less than half a millimeter, thussatisfying the maximum contact lens thickness constraint. Confined to anarea less than half a millimeter across, no existing light emittingdisplay technology can contain multiple millions of pixels that largersize displays now commonly do. This means that multiple small displaysmust be employed to support the desired resolution of a contact lensdisplay.

Definition of term: femto display

Definition of term: femto projector

In this document, the term femto projector, or femto display will referto any display device that can fit into the half millimeter on a sidecube (or less) constraint of fitting into a contact lens, and capable ofproducing the appropriate post-corneal spherical wavefronts. Onespecific instance of such a device is the one described previously, inwhich a planer pixel light emitting element is followed by a positivelens element and then a negative lens element.

It should be noted that traditionally, any self-contained “imageprojector” device emits contracting spherical wavefronts of light thatwill all come into focus at a particular (potentially curved) surface infront of the projector. Light re-emitted from the screen will thenproduce the sort of expanding spherical wavefronts of light that theoutside of the human eye is expecting to receive. While we still use theword projector, femto projectors by definition produce contractingspherical wavefronts of light appropriate for the post-cornealexpectations of the human eye. One difference here is that there is noseparate image forming screen involved.

(The word femto is used, not because of any elements being at thefemto-meter scale, but because the terms micro-projector andpico-projector are already in use defining other classes of largerprojectors.)

Definition of term: femto projector display die

We need a specific term to refer to the (generally planer) source oflight emitting pixels used within a femto projector. In this document wewill use the term femto projector display die for this purpose. Asdescribed in more detail later, this element can be an OLED on silicondisplay, or a LED display, or a LCD display, etc. While the word die isuse in this term, which will later be defined as a type of integratedcircuit, in the general sense intended here the femto projector displaydie does not have to be an IC die, but any small pixel light emittingelement.

VII. Variable Resolution Mappings for Contact Lens Displays

Now that we have an understanding of the variable resolution mapping ofthe retina, we can start considering how the design of a contact lensdisplay can best take advantage of this.

When a Contact Lens is Fixed to the Cornea

If a contact lens display is fixed to the cornea in both position andorientation (not presently common), from the contact lens' point ofview, the varying resolution of the retina is at fixed locations in thedisplay space of the contact lens. That means that the high resolutionfovea will appear at just one location (though offset from the center ofthe contact lens' view), and the contact lens design could be made toonly display high resolution pixels there, and progressively lowerresolution pixels at locations further from the center of the fovea. Andthe “progressively lower resolution” function could take into accountthe different fall-offs in resolution along the different longitudes ofthe retina. This would truly minimize the total number of display pixelsthat a contact lens would have to have, while maximizing the displayedresolution from the eye's point of view.

However, there apparently is some variation of the position of the foveawith respect to the corneal optical axis, and of course there is knownto be at least a three to one difference in the density of cones (squareof resolution) in the foveal maximum cone density zone (e.g., 20/20people vs. 20/10 people (only about 2% of the population though)). Whilethis could be addressed by fixing the contact lens slightly offset tothe cornea, another technique would be to just have a slightly largerhighest resolution zone on the contact lens display.

When a Contact Lens is not Fixed to the Cornea

Given the number of practical problems with rigidly fixing a contactlens to the cornea, what would a contact lens display that is free torotate about the cornea “see” of the retina? First, the center of thefovea, which is nominally offset 5° of visual angle from the cornealoptical axis, could appear at any angle of visual longitude. In such acase, a contact lens display would have to place its highest resolutionpixels everywhere within a circle of 5° of visual eccentricity from thecorneal optical axis, plus a bit more for the width of the highresolution portion of the fovea. Also, because there is a slowerdrop-off in resolution in the nasal direction, but the contact lensdisplay doesn't know which direction that is, it would have to assumethe worst case nasal slower drop off in every direction. That is,outside the greatly enlarged constant resolution foveal display region,the resolution of the contact lens display would have to drop off nofaster than the resolution of the retina drops off from the center ofthe fovea in the nasal direction.

But there is more. A contact lens not fixed to the cornea not only isfree to rotate about it, but also to slide off alignment with thecorneal apex a bit. The effect of this to a contact lens display designis that one has to add another 2° or so of “enlarged” constantresolution foveal display region.

It should be noted that eye mounted displays implanted into the eye arefree of most of these uncertainties, and can be more optimal with howthey parcel out their display resolution. But here we want to teach thebest methods for non-fixed contact lens displays, so we will describehow to construct contact lens displays that can deal with rotation andslippage with respect to the cornea.

It also should also be noted, though, that at the time of “rendering”any particular variable resolution image to be displayed onto a contactlens display, accurate current knowledge as to the orientation and shiftof the contact lens relative to the cornea will always be available.Thus a variable resolution rendering system does not have to render atthe same high fixed resolution all over the polar end-cap; instead itcan do so only where the fovea is actually located during the frame, andcan utilize less rendering resources elsewhere in the majority of theend-cap mapping.

VIII. Means to Display Variable Resolution Pixels

Most all man-made display pixels are all the same size. How are we toconstruct an eye-mounted display, including contact lens displays, inwhich the pixel size is not only variable, but potentially continuouslyvariable?

From the discussion describing the constraints on contact lens displayoptics (as well as that of most other forms of eye mounted displays), weknow that multiple different projectors will have to be employed to formthe desired image on the retina. Thus one potential way of producingdifferent size pixels on the retina would be to have projectors thatproject to successively higher bands of eccentricity on the retina havedifferent amounts of optical magnification, even if all the pixel sizeswithin a given projector would all be the same.

What about the pixels produced by a single projector? Are there any waysto make their sizes change? Optically, one can introduce trapezoidaldistortion, where the width of the projected image on the top is widerthan the width of the projected image on the bottom. Correspondingly,the width of what were square pixels at the display image source planewould be rectangular pixels at the top of the projected image, widerthan high, and less and less wide until the at the bottom of theprojected image they would be square. There are two problems with thisapproach. First, it requires a more complicated tilted optical path thanthe simple two element optical design that was previously described.Second, we not only want the pixels to be wider at the top, but alsotaller, e.g., still locally square.

There is another way in which we can apply an arbitrary mapping functionto the size and expansion of size of pixels: we can just pre-distort theshape of the pixels that are built from square to the size and positionof the pixels of the source image plane device. While modern VLSIfabrication techniques require semiconductor devices to be built asflat, planer objects, the precision of the manufacturing allows pixelsto be built of virtually any shape, so long as the scale of the shape isrelatively large compared to the smallest feature size of the underlyingIC lithography process being used. A figure showing visually what ismeant will be given once the topic of pixel shapes and tilings have beendiscussed.

VIII.A. Pixel Tilings Overview

Most digital devices, both image sensors and displays, are built fromsquare or rectangular, pixel shapes and tilings. But other pixel shapesand tilings, including hexagonal, have been used. The general techniquestaught here can be applied to any form of pixel tiling, specificallyincluding square, rectangular, and hexagonal. But for several reasons,the preferred embodiment is to use hexagonal tilings. As previouslydiscussed, hexagonal tilings are 30% more efficient than square pixeltilings in filling a given area with pixels with a given minimumresolution. In a contact lens display that is free to rotate about thecorneal axis, the pixels will project to the retina at all angles, andthe resolution of square pixel tilings is limited by the length of thediagonal of the square, not the width of the sides. In hexagonal pixeltilings, the resolution is limited by the long width of the hexagon,which is much closer to the width of the short width of the hexagon thanis the diagonal of a square pixel relative to its width. The fact thatthe “pixels of the eye” are also mostly hexagonally tiled is not thereason for our similar choice of tiling of the display source image, butrather an issue of convergent evolution. (Also, hexagonally tiled cellsare the most common tiling of biological systems, but instances ofsquare cell tilings occur too.) There are, however, also disadvantagesto using hexagonal tilings. Square and/or rectangular tilings are easierand thus usually more efficient for computer graphics and imageprocessing algorithms and techniques to utilize than are hexagonaltilings. When red, green, and blue pixels must be produced in a singlesource image plane device, it is quite common to place three one thirdwide by one high rectangular sub pixels together to form a single squareRGB pixel (though other designs, including sideways “W” shaped designs,are also in commercial use). There also is a great deal of existingstandards, technology, and devices designed around square (or at leastrectangular) tilings. But compared to the additional overhead incurredwhen working in a variable resolution space, the additional penalty dueto the use of hexagonal tilings is not great. Because the size of thedisplayed pixels is so closely matched to that of the underlying conesand groups of cones of the retina, other methods of supporting fullcolor display are possible that are compatible with hexagonal pixeltilings, and will be described later.

VIII.B. Projector Tilings

So far we have only been discussing the tiling of individual pixels. Butwe already know that we will have multiple projectors that at theindividual projected image level also have to tile together. As at theindividual pixel level, many different shapes of projected images andtilings of such shapes are possible, specifically including those basedon square, rectangular, and hexagonal projector shapes and tilings, buthere most of the existing art has been utilized square or rectangularprojector image shapes and tilings. The general techniques taught herecan be applied to any form of projector shape and tiling, specificallyincluding those based on square, rectangular, and hexagonal shapedprojector images and tilings. But for several reasons, again thepreferred embodiment is to use hexagonal projector images and tilings.

The idea is to decompose the hexagonal tiling of the rectangularScreenSurface (hexagonally shaped pixels) into a tiling of n-groups ofhexagons (hexagonally shaped projectors). The pixels would be mapped tothe surface of the retina via a locally uniform resolution mapping bytwo techniques, as previously described. First, each ring of projectorswould have a different amount of magnification onto the retina. Second,the pixels that form each projector's image would be pre-distorted tomatch a patch of locally uniform resolution mapping. However, allprojectors would have the same number of pixels, and the mappedresolution will closely match that of the retinal receptor fields of thehuman eye. This is a direct consequence of using the locally uniformresolution mapping. The same will be true for all of the subsequentdifferent parameterizations of projector tilings to be shown.

The decomposition has several parameters and constraints. The choice ofparameters many times will be an engineering tradeoff. The previousparagraph describes the choices we have already made about what mappingto use and shape of pixels and projectors. A constraint based choiceinvolves the separate EndCap mapping: what mapping should be used, whatθ_(min) will be used, how will this mapping image interlock (match) thatof the locally uniform resolution mapping at the boundaries, and what(constant) resolution should be supported? The decomposition intoindividual projectors has choices that are the equivalent of choosing aSW and SH of the tiling of the projectors as hexagonal pixels. The SWequivalent will be the number of semi-column pairs of hexagonally shapedprojectors (pair of an even and odd semi-column) in tiling. This is thesame as the number of projectors per row (even or odd). The SHequivalent will be the number of rows of hexagonally shaped projectors(both even and odd rows) between θ_(min) and θ_(max). When mapped to theViewSphere, each even or odd row of projectors will form a circle ofprojectors around the center of the contact lens at increasing distancesfrom this center. The distances are constrained by where and how largethe entrance pupil for the visual region to be covered by that projectoris. Thus each “row” of projectors will be called a “circle” ofprojectors when viewed after mapping. Thus the parameterization of thedecomposition into projectors has four parameters: θ_(min), θ_(max), thenumber of projectors per circle, and the number of concentric circles.As mentioned before, each ring of projectors will have its ownparticular amount of retinal magnification. Also again, all projectorswill have the same number of pixels (as a direct result of the mapping).The actual number of pixels that every projector must have will be afunction of just the number of projectors per circle and the number ofcircles. Thus a different choice in these parameters will affectengineering trade-offs in the number of projectors and the number ofpixels per projector.

We will illustrate the effects of different choices of parameters fortiling of the projectors by showing several specific combinations ofchoices. FIG. 56 shows 28 hexagonally shaped projectors (aspre-distorted to match a patch of locally uniform resolution mapping),mapped as seven circles of four projectors each. Each group of fourprojectors has the same relative magnification. There are severalcomments that must be stated about this image that will also apply theseveral additional variations that will be described next. First, theimage is an orthographic polar projection of the areas that eachprojector projects to. This means that while shapes near the center ofthe image are represented relatively un-distorted, shapes towards theouter thick edge circle (the silhouette of the ViewSphere) are actuallymore elongated than they look on the spherical surface. Next, thisvariable resolution mapping of projector's images stops short of thenorth pole (the center of the image) to avoid the singularity that isthere. This is the “plus sign” shaped region at the center. This will becovered later by a separate EndCap chart, and will left empty in otherexamples until then. The highest eccentricity that the projectors reachis just shy of 180°, as can be seen by the slight gap between thickcircle representing the silhouette of the ViewSphere and the outercircle representing θ_(max) of this mapping. Also, there are four“half-hexagon” shapes on the outer edge. These represent regions of themapping with an eccentricity below θ_(max), but not covered by any ofthe 28 hexagonally shaped projectors. So the actual portion of theretina covered by the projectors would not include these areas.Alternately, by adding four additional projectors that will only be halfutilized, these four half regions could be included. Examining theoverall mapping of projectors, it can be seen that the outer ring offour cover too large of longitudinal angle to fit the entrance pupil.Also, the shape of the projectors on different rings varies considerablyfrom the inner to the outer ring

Next let us consider what changing to five projectors per ring, stillwith seven rings, would look like. This uses 35 hexagonally shapedprojectors, and is shown in FIG. 57. The portion of the ViewSpherecovered is less than in FIG. 56, even though more projectors are used.This is a direct consequence of using more projectors per ring to coverall longitudes. (The required number of pixel per projector, however,would go down.) But the outer ring of projectors still have to cover toolarge of longitudinal angle.

FIG. 58 shows what happens when there are six projectors per ring. Thisuses 42 hexagonally shaped projectors.

FIG. 59 shows what happens when there are seven projectors per ring.This uses 49 hexagonally shaped projectors.

FIG. 60 shows what happens when there are eight projectors per ring.This uses 56 hexagonally shaped projectors.

Under different circumstances, all of these possible projector tilings,as well as others with more or less rings, are of potential use.

EndCap Projector Tilings

For the EndCap mapping, constant resolution projectors will be used.There is a good match between hexagonally tiled constant resolutionprojectors and tilings of variable resolution projectors when the numberof projectors per ring is six. This way the six-way symmetry of thefixed resolution hexagons “matches” the six-way symmetry of the variableresolution wrap around ring. This is illustrated by first looking at thepolar hole in the variable resolution projector mapping, which is shownin FIG. 61 element 6110. This polar hole is due to setting θ_(min) to avalue above zero, in this case about 6°. Otherwise there would have beenan infinite number of projectors spiraling in smaller and smaller intothe north pole. Now we superimpose over the polar hole six fixedresolution hexagonally tiled hexagonally shaped projectors, shown inFIG. 61 as element 6120. Notice that there is very little overlapbetween the fixed and variable resolution projectors, yet the entiresurface is covered. Note also that the first circle of six variableresolution projectors are similar in shape and size as each of the sixfixed resolution projectors. So we can expand the EndCap fovealresolution area by replacing these with six more foveal projectors, ifdesired. This is shown in the overlap image of FIG. 62. The same sort oflevels of overlap can be applied to variable resolution tilings with adifferent number of projectors per ring, but they won't be as efficient.Since the size (and pixel density) of the fixed resolution projectors isnot fixed directly to the size of the variable resolution projectors,their relative scale becomes another design parameter, though the idealis to have similar pixel densities at the joint boundaries.

Given that a ring size of six seems a good number, with the 13 fixedresolution foveal projectors and 30 variable resolution peripheralprojectors, and a θ_(max) above 65°, FIG. 63 shows a preferredembodiment.

If a larger field of view is desired, more than seven rings can be used.The next two figures shows what this looks like (though back to a sevenfixed resolution EndCap).

FIG. 64 element 6410 shows what an eight ring tiling with a seven fixedresolution projector EndCap looks like (49 total projectors).

FIG. 64 element 6420 shows what an eight ring tiling with a seven fixedresolution projector EndCap looks like (55 total projectors).

FIG. 65 element 6510 shows what an eight ring tiling with a seven fixedresolution projector EndCap looks like (61 total projectors).

FIG. 65 element 6520 shows what an eight ring tiling with a seven fixedresolution projector EndCap looks like (67 total projectors).

The last configuration has a (circular) field of view of nearly 180° !The main problem with these additional rings is that their longitudinalsubtended angle keeps getting higher and higher while the maximumlongitudinal subtended angle entrance pupil keeps getting smaller andsmaller. Depending on other optical and efficiency factors, at somepoint the far peripheral rings would have to transition to a tiling madeup of more projectors, but with a smaller number of pixels, andtherefore subtended longitudinal angle, once again fitting within theforeshortened entrance pupil of the eye. There is not a hard cut-off,such as at the point where the entrance pupil becomes narrower than theexit pupil of the projector assigned to that region of the retina. Thisis because at the eye's pupil, the image is still well out of focus, sothe effect will at first be a vignetting of the projector's image on theretina, and an associated loss of the total amount of light from theprojector that will make it through the pupil, lowering the overallillumination power of the projector. This sort of effect can becountered by having a brighter overall projector with a pre-computeddimming of the center of projection, to pre-correct for the vignettingeffects. If, however, a third tiling is needed, then it can be anotherhexagonal tiling of projectors mapped by locally uniform resolution, itjust will have to have a larger number of projectors per ring. Whatwould this look like? That's partially why FIG. 59 and FIG. 60 wereincluded, they show the general form of having more projectors per ring.

As mentioned before, the entrance pupil of the eye also restricts thelocations on the contact lens at which these tiled projectors can beplaced. It is not too restrictive for the central projectors, butbecomes a tighter and tighter restriction for higher visual eccentricityprojectors up to the point where the pupil starts becoming too small tocapture the whole display image without excessive vignetting. To see theeffects of these constraints on out preferred embodiment of 13 fixedresolution foveal projectors and 30 variable resolution peripheralprojectors, FIG. 87 shows one set of placement choices that satisfiesthe constraints while trying to keep the projectors as far apart aspossible. The larger black circles are the fixed resolution projectors;the smaller ones the variable resolution ones. FIG. 88, shows the sameimage as FIG. 87, except this time inside each black circle there is aprojector number starting with one at the center. In both figures theouter circle is the edge of the optical zone of the contact lens, about8 millimeters in diameter. The fixed resolution projectors are less thanhalf a millimeter across; the variable resolution projectors are lessthan a third of a millimeter across.

Projector Tiling Overlap

One issue that was not directly addressed in the proceeding discussionswas that of building a certain amount of “redundant” overlap between theprojected images of projectors on the retina. If all manufacturing andoptics was perfect, no such would be needed. But in the slightlyimperfect real world of optical design trade-offs and fabrication andassembly tolerances, as will be discussed in more detail later, there isa need to have a certain amount of such built-in overlap betweenprojectors. The easiest way to think of how this effects the details ofthe tilings just discussed is to assume that each ring of projectors inthe existing tilings has its common ring retinal magnification amountset 5 to 10% higher than it otherwise would be. The exact amount ofoverlap required is a trade-off between the extra costs and hard limitsof fabrication and assembly technology (as well as the realities ofoptical design) versus the costs incurred by having to build more pixelsthan would otherwise be used. Fortunately the tilings described areeasily amenable to such magnification modifications.

VIII.C. Pixel Tilings Details

As previously described, the projectors use hexagonal tilings ofhexagonal shaped pixels. The complete shape is that of an n-group ofhexagons on their ends, which looks like a single hexagon on its side.In the projectors used for the fixed resolution EndCap mapping all thepixels are the same size, resulting in a uniform tiling. One engineeringdriven constraint on projectors is that less hardware is used if thenumber of pixels semi-columns is equal to or just less than a power oftwo. Since an n-group of hexagons has 2·n+1 hexagons across its widestrow, we will illustrate choices of n for which n is one less than apower of two: 3, 7, 15, 31, and 63.

FIG. 66 shows a 3-group 7 hexagons wide fixed resolution 37 pixelprojector.

FIG. 67 shows a 7-group 15 hexagons wide fixed resolution 169 pixelprojector.

FIG. 68 shows a 15-group 31 hexagons wide fixed resolution 721 pixelprojector.

FIG. 69 shows a 31-group 63 hexagons wide fixed resolution 2,997 pixelprojector.

FIG. 70 shows a 63-group 127 hexagons wide fixed resolution 12,097 pixelprojector.

Currently the 12,097 pixel projector is the choice of a preferredembodiment for the fixed resolution, foveal projectors, but this issubject to other engineering trade-offs.

For the variable resolution peripheral projectors, many times a lowernumber of pixels per projector is needed than for fixed resolutionprojectors. So we will illustrate the variable resolution projectors forvalues of n of 5, 10, 20, and 40.

FIG. 71 shows a 5-group 11 hexagons wide variable resolution 91 pixelprojector.

FIG. 72 shows a 10-group 21 hexagons wide variable resolution 331 pixelprojector.

FIG. 73 shows a 20-group 41 hexagons wide variable resolution 1,261pixel projector.

FIG. 74 shows a 40-group 81 hexagons wide variable resolution 4,921pixel projector.

Currently the 4,921 pixel projector is the choice of a preferredembodiment for the variable resolution, peripheral projectors, but thisis subject to other engineering trade-offs.

It can be seen that the width of an individual hexagonal shaped pixelchanges by almost a factor of two across height of the projector. Thisis one of the engineering reasons why that in general there is aconstraint to have less pixels in total on variable resolutionprojectors than for fixed.

VIII.D. Sub-Sampling of Color for an Eye-Mounted Display

Color pixels have traditionally been one red, one green, and one bluepixel. However, the color cones of the eye are not evenly distributed:on average, there appears to be twice as many red cones as green cones,and four times as many green cones as blue cones. But there isconsiderable individual variation, and the different color cones are notevenly distributed. Because the individual pixels of a contact lensdisplay are shifted with time over an average cone integration time,individual pixels can be a single color each. The color of pixels can besomewhat randomly distributed, with, for example, four red pixels andfour green pixels for every blue pixel. Many other similar combinationsare possible, and there doesn't even have to be a repeating pattern. Themain constraint is that the pixel component generation sub-unit has toknow which of the three primary colors each individual pixel in adisplay is.

VIII.E. Computer Graphics Real-Time Variable Resolution Rendering

There are many different techniques for rendering known in computergraphics; while they traditionally have been designed to render to(approximately) fixed resolution pixels of the ViewPlane, most can beadapted to render into a pixel space that has a quite non-linear mappingto the ViewSphere, specifically including the locally uniform resolutionmapping.

One class of computer graphics rendering techniques is that of raytracing. While traditionally rays are traced from the EyePoint throughthe center of each pixel on the ViewPlane, the rays to be traced caninstead be formed by rays from the EyePoint through the center of apixel mapped back to the ViewSphere by an arbitrary mapping from aScreenSurface. In a preferred embodiment, the mapping is that of locallyuniform resolution. (It is easier when the mapping preserves the pixelstructure, such as OrthogonalLongitudeEccentricity mappings.) To performanti-aliased ray tracing, additional rays at sub-pixel locations arespecified and traced for each pixel.

Another class of computer graphics rendering techniques is that ofincremental z-buffered rendering of (both 2D and 3D) geometric graphicsprimitives. Such primitives include, but are not limited to, two andthree dimensional points, lines, triangles, and higher order surfaces,such as subdivision surfaces, and NURBS, as well as volumetricprimitives. Additional geometric detail is often added to suchprimitives by use of displacement mapping of the surfaces. In thespecial case where the ScreenSurface is the ViewPlane, lines andtriangles in 3D space project into lines and triangles on the ViewPlane.This simplifies the rendering of such primitives, as they can berendered by simple linear interpolation on the ViewPlane. However,higher order surfaces need to be tessellated by some means into smalltriangles, which typically are less than a pixel in size. When renderingto more complex ScreenSurfaces, such as that of locally uniformresolution, rendering of lines and triangles cannot be correctlyachieved by linear interpolation on the ScreenSurface. However, ifhardware is available that can in real-time tessellate higher ordersurfaces into sub-pixels triangles, it trivially can do the same forplaner 3D triangles. Thus the curved nature of these more complexScreenSurfaces is no longer an issue, as once primitives have beentessellated in ViewSpace (or an equivalent space) such that theprojection of the tessellated triangles produced are all less than apixel in size on the ScreenSurface, the difference between a planerinterpolation of the sub-pixel size triangle and the curved sub-pixelportion of the ScreenSurface is vanishingly small.

As an example of how such a tessellation processes into a highlyvariable resolution space, such as that of locally uniform resolution,would work, the example of tessellation of subdivision surfaces viasubdivision will be worked out in detail.

Subdivision surfaces include those such as the rectangular one ofCattmul-Clark, and the triangular Loop model. We will describe thesubdivision process recursively, though actual efficient hardware andsoftware implementations process the work in a more efficient order, asis known to those skilled in the art. An essential element of thesubdivision process is the edge subdivision criteria. Given two verticesin ViewSpace that represent the endpoints of the edge, the subdivisioncriteria will say whether this edge should be subject to furthersubdivision, or not. This is a criteria based only on the informationfrom one edge within the surface, e.g. just the ViewSpace location andattributes of the two endpoint vertices of the edge. The limitation tothis edge-only information for the subdivision criteria, e.g., nottaking into account information about any additional edges that areconnected to one or the other of the two vertices, allows the decisionto subdivide an edge, or not, to be computed anywhere the same edgeoccurs, without having to have explicit links between differentsubdivision surfaces that may contain the same edge.

There have historically been many different edge subdivision criteria,applied either in ViewSpace or on the ScreenSurface, including maximumcordial deviation allowed, a maximum edge length allowed, etc. Here wewill define two such example criteria: one is the maximum length of anedge on the ScreenSurface, the second is the maximum length that an edgewould be on the ScreenSurface if its orientation in ViewSpace had beenparallel to the local normal to the ViewSphere, rather than its actualorientation in ViewSpace. The second criteria keeps long thin trianglesfrom being produced near silhouette edges of the subdivision surface,and is especially important if the tessellated subdivision surface is tothen subject to displacement mapping.

The length of the edge on the ScreenSurface subdivision test can bedescribed in detail as follows. First, the two three-dimensionalvertices of an edge in ViewSpace (or an equivalent space) are mapped totwo two-dimensional (2D) vertices on the ScreenSurface (using thevarious equations already developed). Before taking a measure of thedistance between these two 2D vertices, because many ScreenSurfaces wraparound at the ends of the u range, the two vertices need to be put intoan un-wrapped space. Assuming that u1 and u2 are the u coordinates ofthe two ScreenSurface vertices of the edge, the following pseudo-codefragment defines the un-wrapping procedure:

if (u2>u1 && (u2−u1)>((u1+SW)−u2) then u1=u1+SW

-   -   else if (u1>u2 && (u1−u2)>((u2+SW)−u1) then u2=u2+SW

After this, the distance between the two (unwrapped) points on theScreenSurface can be taken using the distance metric of choice, e.g.Manhattan, Euclidian, Euclidian squared, etc. This distance can then becompared to a (fixed) maximum distance threshold on the ScreenSurface:if it is below the threshold the edge will not be subject to furthersubdivision; if the edge is equal to or above the threshold, then theedge will be further subdivided. The test for the case when the edgelength as projected as if it were oriented parallel to the local normalto the ViewSphere is a straightforward variation of the one given.Although the example threshold of “being smaller than a pixel” has beenused earlier, in the general case represented here, the length thresholdcan be set to any of a range of values, including those actually largerthan a single pixel.

However, knowing which edges of a quadrilateral or a triangle should besubject to further subdivision is only the starting part of thealgorithm to decide how to subdivide the quadrilateral mesh or atriangle mesh. Fortunately, and this is a great advantage of thesubdivision test on the ScreenSurface as described, the details of themesh sub-division process is the same for the case when we are using thelocally uniform resolution mapping as when using the more historicalViewPlane mapping, and thus need not be further detailed here. Theresultant images, however, will be quite different. Clipping ofsubdivision surfaces works a bit differently when using highly variableresolution, as while most portions of the subdivision surface outsidethe truncated view frustum (or, more accurately, truncated view cone)can be discarded before being subject to much sub-division, someportions will need to be subdivided down to the pixel level to supportadjacent portions that are within the view cone. Similar comments applyto the front and rear clipping planes.

It should be noted that more sophisticated subdivision surfaceprimitives include additional subdivision surfaces for texturecoordinates, etc. These, however, can be extended in the same way as thepositional subdivision was.

In general, any tessellation process that doesn't utilize equal-spacedtessellation, will have a more complex job in computing localderivatives of various quintiles, such as texture address derivatives.

Once a subdivision surface (or even just a planer 3D triangle) has beensub-divided into sub-pixel size triangles, the individual vertices ofthe triangle, known as micro-vertices, are subject to the standardprogrammable pixel shading process, which may first include displacementmapping.

Once the (possibly displacement mapped offset, and possibly then subjectto more tessellation to get the micro-triangles back down below thedesired size) vertices of the sub-pixel size triangles have had color,depth, alpha, and possibly other attributes assigned, the individualtriangles can be linearly interpolated on ScreenSurface to the sub-pixellocations of the local super-sample points in the pixels touched, andthe results z-buffered into each sample that ends up being covered bythe triangle. (When supporting motion blur and/or depth of field, thealgorithm is a bit more complex, but otherwise similar to that wellknown in the art.) The resultant super sample buffer will itself reflectthe highly variable resolution of the highly variable resolution mappinginvolved. But this is the desired results.

The above has described the tessellation via subdivision of subdivisionsurfaces; similar approaches can be applied to the differenttessellation approaches of other higher order surface types and ofvolumetric primitives.

While the above has described the general case of tessellation of higherorder surfaces, the tessellation of simpler geometric primitives, suchas curved lines, straight lines, points, and anti-aliased versions ofthese is a simplified sub-set of the above process.

Another alternative class of rendering algorithms is similar to the onejust described, but the programmable shading portion of the process isdeferred until after all the z-buffering has taken place. This has theadvantage that only visible pixels have to have the programmable shadingsub-program run on them, but at the expense of fairly extensivebookkeeping to keep track of the shader context as triangles areproduced for z-buffering. This general technique is known as hardwaredeferred shading [Deering, M., Winner, S., Schediwy, B., Duffy, C andHunt, N. 1988. The Triangle Processor and Normal Vector Shader: A VLSIsystem for High Performance Graphics. In Computer Graphics (Proceedingsof SIGGRAPH 88), 22 (4) ACM, 21-30.].

The final stage of the rendering process is the conversion of thesuper-sampled buffer into discrete pixel component values for the targetdisplay device, described in the next sub-section.

VIII.F. Generation of Discrete Pixel Component Values from theSuper-Sample Buffer

Once a sample buffer has been generated for a whole scene of rendering,the sample buffer is converted to discrete pixels for display byapplication of a convolution of a pixel filtering function centered onwhere each desired final pixel is to be generated from, with the extent(size) of the filter proportional to the area of the final pixel, thoughthe constant of proportionality depends on the choice of filter. Thesimplest filters are square box filters, with the extent set to be thesame as the final pixel size. (For hexagonally shaped pixels, ahexagonally shaped “box” filter is the most appropriate, but a circularfilter will work well too.) The “theoretically” most correct filter isthe infinite sync function, which has an infinite extent. The highestquality filter in common use in software renderers is either an 11×11 ora 9×9 pixel size windowed sync filter. The highest quality filterimplemented in hardware is generally considered to be the 4×4 Mitchelfilter, though the hardware involved can implement any circularlysymmetric 5×5 filter [Deering, M., and Naegle, D. 2002. The SageGraphics Architecture. In Proceedings of SIGGRAPH 2002, ACM Press/ACMSIGGRAPH, New York. Akeley, K., Ed., Computer Graphics Proceedings,Annual Conference Series, ACM, 683-692.]. A 5×5 filter extent is neededin the general case to implement a 4×4 filter when the center locationof the final pixel to be produced can be at a location other than thecenter of a super-sampled pixel.

Such convolution filters are seen as simultaneously performing twofunctions: an interpolated re-construction of the underlying imagefunction, and a low-pass band filter to remove frequencies higher thanare supportable by the Nyquist rate of the final pixel pitch. In thenatural world, no such low pass filter occurs before the eye, why? Inthe natural world, (primarily) the limited quality optics of the eyeonly pass spatial frequencies below or near the peak Nyquist rate of theunderlying photoreceptor mosaic. This doesn't work for most manmadedisplays, as in general the distance to the viewer, and thus the spatialfrequencies produced on the retina, are not known. However, foreye-mounted displays (and head-mounted displays) the distance size thatdisplay pixel subtend on the retina is known. For head-mounted displays,however, which portion of the retina is being displayed to generallyisn't known, so the relative size of the display pixels to the retinalreceptor fields isn't known. But this is known for eye-mounted displays.This means that, in principle, the amount of low-pass spatial filteringthat will be caused by the (remaining) optics of the eye (as well as theoptics of the projectors) can be known. Blindly applying a traditionalanti-aliasing filter can result in an over filtering of the imagedisplayed. Practically, the spatial low pass filtering caused by theeye's (remaining) optics is individual specific, though it is easilydetermined: to a first approximation it is given by the viewer's maximumacuity (e.g., 20/20, etc.). Thus, depending on the relative size of thedisplayed pixels to the viewer's spatial frequency perception limit(optically dominated in the fovea, midget ganglion cell visual fieldcenter size dominated in the periphery) the appropriate amount ofspatial frequency filtering that should be performed by thesuper-sampled pixel region to display pixel value can be computed, andused to drive the size (and shape) of filtering actually performed, asparameterized by visual eccentricity, of super-samples to final displaypixel values. Note that in the limit when display pixels aresufficiently smaller than the midget ganglion cell visual field centersize, the correct filter can be the box filter! Again, in practice, thetrade-off between aliasing and sharpness is an individual preference,and user level “sharpness” controls should be available to allow theindividual user to tune it to personal taste. There is also a contentbias to this setting; for antialiased lines, people prefer a lowerspatial frequency cut-off, e.g., nearly complete elimination of“jaggies,” but at the expense of a slightly blurrier line, whereas fortext, the preference is for a higher spatial frequency cut-off thatpreserves sharpness, e.g., serif's in fonts.

VIII.G. Computer Graphics Real-Time Rendering of ArbitraryPre-Inverse-Distortion of Pixels

General classes of computer graphics rendering algorithms leave theirresults as a super-sampled image on a ScreenSurface. The finalconversion of this data for output to the video display device is aconvolution filter that takes in a region of samples and outputs a colorcomponent for display. In the simple case, the location of the center ofthis region is just a simple x-y scan of the sample buffer. In a morecomplex implementation, the location of the center of each region can beperturbed by a limited amount specified by another function unit. Thisfunction unit may be interpolating a spline function, or directlylooking up a x-y perturbation value for each pixel to be output to thedisplay. In this way, arbitrary pre-inverse-distortions of the finalanti-aliased output image can be performed. Because at least some of thedistortions will be slightly different for each color component value,each color component needs to have its own unique perturbations suppliedto it. In the special case in which there is only one color componentper pixel displayed, only one perturbation per output pixel need becomputed, but it has to be the perturbation that is correct for theparticular color component presently being displayed.

Technically, if the pre-inverse-distortion involves a change in scale,e.g., a magnification or minification, then the size of the region ofsuper-samples and the associated anti-aliasing convolution filter needsto be changed in a like way. However, if the extent of the change inscale is limited to a few percent, such additional changes may notaffect the results sufficiently to be justified. In general, perturbedconvolution units can be constructed in either way.

VIII.H. Using Computer Graphics Real-Time Rendering of ArbitraryPre-Inverse-Distortion of Pixels to Compensate for a Varity of Opticaland Manufacturing Based Distortions

The two lens per projector display mesh design described will have atleast two types of residual optical distortion inherent in the design,and several more likely sources of distortions due to un-avoidabletolerance errors inherent in the fabrication and assembly manufacturingprocess. The impact of all of these errors on the image quality at theretinal surface can be minimized or eliminated by use of the previouslydescribed pre-inverse-distortion of pixels by the computer graphicsrendering sub-system. This sub-section will list these likely andpotential sources of distortion.

Residual chromatic aberration inherent in the optical design: the simpletwo lens design has a residual chromatic aberration component largerthan that of most commercial optical designs. The “normal” fixes, e.g.,use of doublets, or GRIN lenses, and are generally not practical for thesmall amount of projection optics space available. However the residualchromatic aberration can be greatly reduced by the appropriate colorcomponent specific pre-distortion of the image by the sample filteringsub-system.

Residual spherical aberration inherent in the optical design: the simpletwo lens design has a residual spherical aberration component; thisappears as pin-cushion or barrel distortion on the image projected ontothe surface of the retina. The amount of such distortion is limitedbecause the display surface is spherical (the inside of the retinalsphere), but some remains because the image production device is(necessarily) planer. Again, this distortion can be greatly reduced oreliminated by the appropriate pre-distortion of the image by the samplefiltering sub-system.

Error in the amount of magnification produced by the optics due tooptical lens fabrication variation: the projector lenses as actuallyfabricated will differ in the amount of magnification produced on theretina from the ideal design, due to un-avoidable tolerance errorsinduced by the fabrication process. There is a trade-off between thecost of fabrication and the magnitude of such errors caused by thefabrication process, and there may be a lower practical limit to how lowthe errors can be reduced to. By building more pixels on the displaydevice than needed in the ideal case, additional magnification can beprovided to compensate to any lost in the fabrication process, againthrough the appropriate pre-distortion of the image by the samplefiltering sub-system. In this case the pre-distortion is just to spreadout the image to be displayed over more pixels on the display than wouldotherwise be the case. Compensating for too much magnification caused bylens fabrication issues can be corrected for in the reverse way.

Errors caused by offset in the lens(es) center position due to eitherthe fabrication or assembly process: slight decentering of the optics asbuilt will cause a corresponding offset of the position of the displayedimage on the retina. Once again, by having extra display pixelsavailable at the outer edge of the display device, these sort of errorscan be greatly reduced or eliminated by the appropriate pre-distortionof the image by the sample filtering sub-system. In this case thepre-distortion required is the appropriate (simple) shift of where theimage is displayed on the display device, though care must be taken tomake sure that the desired region of the retina is still projected to.

Errors caused to tilt in the lens(es) due to either the fabrication orassembly process: slight tilt of the as built optics will cause acorresponding elliptical scaling of the displayed image on the retina.Once again, these sort of errors can be greatly reduced or eliminated bythe appropriate pre-distortion of the image by the sample filteringsub-system. In this case the pre-distortion required is the appropriatenon-uniform scaling of the image displayed on the display device.

Other errors due to either the fabrication or assembly process: Thereare other types of distortions of the image as displayed on the retinadue to manufacturing issues. In general, most all of these sort oferrors can be greatly reduced or eliminated by the appropriatepre-distortion of the image by the sample filtering sub-system.

Flatness of field and Vignetting: Another common distortion of opticalsystem is an un-evenness in the intensity of the image at all pixels. Insome cases where there are pupil limits within the overall optical path,vignetting can occur. This class of errors can be greatly reduced oreliminated by the appropriate emphasis of the brightness of individualpixels of the image by the sample filtering sub-system. In general,there will be a per display pixel intensity correction factor applied.This can include not only correcting for vignetting effects, but alsocan per pixel correct for manufactured variations in actual pixelintensities produced by a given physically constructed pixel, as well ascorrect the intensity of light output from variable resolution pixels tothe correct values.

“Feathering” between the edges of adjacent projectors: this is theproblem in matching the edges of adjacent projectors while alsocompensating for all of the above optical distortions. While theappropriate pre-distortion of the image to be display may effectivelyeliminate all of the potential optical distortions described, the shapeof the overall projected image and of the individual pixels on theretina will still be distorted relative to the “ideal.” The appropriatecorrected pixel display values will be displayed, eliminating the effectof the distortion, but the distortions from one projector to itsneighbors will not in general be the same. Thus two projectors thatshare an edge will not meet at perfect pixel boundaries. “Edgefeathering” is a technique that addresses this problem. Here, onceagain, the display devices are assumed to be slightly over-provisionedin pixels, so that there will always be at least a one pixel overlapbetween the displayed image on the retina of one projector to anyprojector that shares an edge with it. By pre-computing how much eachpixel contributes to the overlapped area, each pixel component value canhave its intensity diminished such that the overall intensitycontributed by both pixels to the overlap is similar to what theintensity should had been if only one pixel was displaying to the area.The intensity is “similar” because each pixel also has to display partof its area to a non-overlapped portion of the retina. While in somecases this “feathering” may take place over a number of pixels ofoverlap, to reduce the amount of over-provisioning of pixels in thedisplay device, the feathering can be reduced to nearly a single pixelof overlap. There are several additional techniques to help with theoverlap. First the overall display rate is higher than the lightintegration time of the cone cells, so that the overlapped pixels willbe displayed several times over the time interval that the cones areintegrating light. This means that because of (likely) tracked slippageof the contact lens on the cornea (for a contact lens based eye mounteddisplay), the retinal location of the overlap will change several times,reducing its effect. Another effective technique is the use ofsub-pixels in the region of the display near the edges. Turning on andoff sub-pixels allows for closer matching of the pixels between twoadjacent projectors before feathering is applied. At the same time,because such sub-pixels are individually addressed only for the outeredge of the display device, the amount of pixel (and sub-pixel) datathat has to be sent to the device is only slightly increased. Finally,while the vast majority of overlapping pixels occur on the boundary edgebetween just two projectors; at the corners where three projectors meetin a hexagonal projector tiling, or four projectors meet in arectangular projector tiling, three (four) projector pixels have to allbe feathered together. The feathering intensity reduction can be handledby the same super-sample region to pixel component sub-unit thatperforms the pre-distortions of the image, because it generally willhave knowledge of where the overlapping edges between projectors are.

The general solution to producing the desired high quality image on thesurface of the retina consists of four parts. First, limiting themaximum amount of errors by appropriate design of the opticalcomponents, and tight tolerances in the fabrication and assemblyprocess. Second, the over provisioning of pixels (and optical fieldarea) in the display. Third, the accurate per-manufactured deviceidentification of the combination effect of all the errors present. Andfinally, the appropriate programmed pre-distortion of the final pixelsto be displayed by the super-sample region to pixel component sub-unit.Important note: all the required pre-distortions can be combinedtogether into a single distortion vector per output pixel component. Asmentioned before, this distortion field can either be explicit per pixelcomponent location center of convolution offsets, or based on some formof spline interpolation or a similar process. The later can be performedon hardware in real-time from a relatively small number of distortioncomponents, small enough that they can be stored on chip, an explicitvector for every pixel to be output requires enough storage that it maybe better streamed in from external RAM every sub-frame.

There is another source of “error”: as described before, there isindividual variance in the retinal radius of people's eyes. (E.g., eacheye in the pair are generally quite similar, but can differ in size fromother individuals.) This will show up as a different magnification ofthe projected images onto the surface of individual's retinas. Some ofthis is corrected for by individual specific corrective optics in thefinal portion of the projector optical design, but there will still besome residual error in magnification. If the error were large enough,there could also be an error in focus, but that is generally not thecase. The residual individual specific magnification error can becorrected for by the same techniques as have been previously describedfor other classes of optical errors, once the magnitude of error hasbeen empirically determined.

We have used the term “pre-distortion of pixels” (actually,pre-inverse-distortion) as a general mechanism for correcting forinherent optical design limitations, errors caused by the need for somelevel of tolerance in the fabrication of optics and other parts, errorscaused by the need for some level of tolerance in the mechanicalassembly of optics and other parts, errors caused by the need to abutdifferent projector's images on the retina, and errors caused by usingthe same optical design for people with slight individual variation ineye size, etc. Only errors purely of intensity are corrected for by acompanion correction. We so far have defined the method to cause desireddistortion in pixels as the appropriate convolution operation on thehigher resolution (but also highly aliased) sample buffer, as producedby many computer graphics rendering algorithms. This is indeed thepreferred embodiment, as it produces the highest quality final pixels.However, there are simpler, but lower quality methods of producingdistorted pixels. Rather that starting with a super sample buffer, onemight instead have an image at the pixel level. One could then stillcompute the intensity value for pixel components that would appear at asub-pixel offset between the original pixels either via convolution, orsimple interpolation (which is really a simplified form of convolution).The site of the interpolated pixels can also be approximated, allowingan alternate mechanism for matching the inherent variable resolution ofthe retina. Indeed even when the method of convolution of a super samplebuffer is employed, the original super-sample values may have beengenerated by traditional texture mapping techniques from a texture thatwas a fixed resolution pixel image, such as a frame of incomingtraditional video. All of these methods of producing the final pixelcomponent intensity values are subject to trade-offs in design, andcompatibility issues with existing content.

VIII.I. Issues of Accommodation and Presbyopia

So far all the optical errors have been passive errors, and thus thecorrections can be performed by “passive” mechanisms, e.g. the renderingpre-distortion process. There is an active source of optical changesthat it would be desirable to correct in future versions of eye-mounteddisplays. The source is changes in the focus of the eye's lens due toaccommodation. The effects are different in the two optical paths:through the normal outside world imaged onto the retina by the bulkcontact lens (independent of any images being displayed), and thedisplayed virtual image through the optics of the display projectors. Asdescribed so far, the virtual image produced by the display will be infocus at a relatively fixed distance from the observer. Due to being astereo display, portions of the image will be closer or further awaythan the plane of perfect focus. This is a general issue for most stereodisplay systems. In an individual with advanced presbyopia, this is nota problem, as they have effectively only one distance at which objectswill be in focus. Their problem is the general problem of those who havepresbyopia: real world objects at different distances than the singleplane of perfect focus will be relatively out of focus; this can only bechanged by use of reading glasses or bifocals. In an individual withlittle or no presbyopia, the accommodation mechanism ensures that whenthey converge their eyes on portions of the virtual display in front ofor behind its plane of perfect focus, their eye's lenses will changefocus and cause the virtual image to be out of focus at such points,though real world objects at such locations will be at the correctfocus.

For individuals with little or no presbyopia, if the display projectorsoptics had some ability to change the distance at which they are infocus, then by observing the vergence angle between the two eyes, thevirtual image being displayed could be made to come into optical focusat the same distance that the individual viewer's eyes are dynamicallyfocused. (Technically the renderer should only render sharp images forthose portions of the virtual world that is relatively near the currentdynamic plane of focus; objects further away to either side should berendered as blurry to match what happens in the physical world.) Thisdoes the trick for non-presbyopic individuals. Because the lenses in thecurrently proposed preferred method projectors are so small (less thanhalf a millimeter across), it is quite feasible for such changes inprojector focus to be caused by either piezoelectrical elements, or MEMSelements built within the projector. Other methods include those thatdirectly affect the curvature of the projector lenses.

To improve life for individuals with presbyopia, the opposite is needed:the focus through the optical path through the bulk contact lens needsto be slightly changeable in focus. Since the light from the physicalworld that forms images on the retina is that which passes through theregions of the contact lens between the display projectors, if focusaltering elements are placed between the display projectors, then againby observing in real-time the vergence angle between the individual'stwo eyes (which any eye-mounted display has to do anyway), theappropriate correction of focus can be dynamically applied to thethrough the bulk lens optical path. This “cures” presbyopia even when novirtual image is being displayed! Again because of the relatively smallsize of the lenses involved that would be placed between the displayprojectors, either piezoelectrical elements or MEMS elements should beable to cause the desired range of dynamic focus.

In the most general case, if the range of diopters of focus that can bedynamically programmed into the bulk contact lens path is large enough,it is possible to have programmable contact lenses. This would be asolution for people with either myopia or hypermetropia. This would be atruly “one size fits all” vision device. Given the ability of a contactlens display to directly image the surface of the retina, such devicescould also be auto-prescribing. You put them on, and they just “work.”However, given the consequences of failure, especially while anindividual is driving a motor vehicle, any such mechanism may includesafety locks and features to enhance reliability.

IX. Physical Construction of the Display Mesh Terminology

In both the technology of integrated circuit chips, and display devices,their ambiguities in terminology. This section more precisely defineswhat certain terms are meant to mean in this document.

Definition of term: integrated circuit

Definition of term: IC

An integrated circuit (or IC) is generally a single physical objectonto/into which a large (to very large) number of electrical circuitshave been assembled. While traditionally electrical circuits are thoughtof, today the circuitry can also be optical and/or mechanical. And themechanical circuits (such as MEMS) doesn't move just small portions ofsolid objects, liquids can be moved (or induce movement) as well.Optical circuitry can not only either receive or produce light (or moregeneral electrical magnetic radiation), but also process light directly.Will the most common substrate for ICs is silicon, a variety of othermaterials, such as Gallium Arsenide, are in use today. And even thoughthe substrate may be comprised of one material, during circuitfabrication, other materials, such as aluminum, or light emittingorganic compounds, are frequently deposited on top of the substratematerial.

Definition of term: die

Definition of term: bare die

Definition of term: passivated die

Definition of term: hermetically sealed die

The single individual IC physical object not including an additionalpackaging is commonly called a die, or a bare die. It is common for thetop and sides of a die to be covered with a protective coating toinsulate the sensitive materials of the electronics from the negativeeffects of oxygen, water, and other possible harmful containments. Manytimes this encapsulation meets a technical definition of hermeticsealing (simplified meaning protected from air or gas), in which casethe die can be considered a hermetically sealed die. Here the term baredie will be used to always have these meanings, implicitly includingsome form of hermetic sealing when this is required in context. Itshould be noted that many forms of such sealing can be opticallytransparent.

Definition of term: wafer

Definition of term: IC wafer

Typically integrated circuit die are not fabricated one at a time, butinstead a large number of them are fabricated at the same time on alarger (usually circular) portion of the substrate material called awafer, or an IC wafer. After (most) fabrication, the wafer is diced upinto individual (usually identical) rectangles (or other shapes) whichare the bare die.

Definition of term: dicing

Definition of term: diced

While historically the processes of dicing an IC wafer into individualdie involves cutting the wafer up along rows and columns, producingrectangular shaped die, it is now possible for individual die to befurther trimmed into other shapes, such as hexagonal or octagonal.(Hexagonal is a common shape presently for high power LED die.)

Definition of term: wafer thinning

Definition of term: die thinning

Because wafers are by definition much larger than die, e.g. 6 to 12 to18 inches in diameter, they usually start out as a fairly thick piece ofthe substrate material (0.4 to 1.1 millimeters) just for structuralsupport reasons during fabrication. But the active electrical circuitryis contained only in the top two dozen microns or so of the wafer. Forvarious different reasons, this can be too thick. One example is when anoptically transparent integrated circuit is desired, such as a microLCD. Another is when it is desired to place the electrical interconnectsto the die on the rear side of the die. Other times there is very littlespace for the integrated circuit. In all these cases, a much thinner diethan standard can be achieved by “thinning” excess material off therear, allowing final thicknesses to be in the 20 micron range. While theis commonly called wafer thinning, and the thinning can be achieved atthe wafer level before dicing, thinning of the individual die afterdicing, or die thinning, is also possible.

Definition of term: chip

Definition of term: IC chip

Definition of term: integrated circuit chip

Definition of term: packaged chip

When an integrated circuit is wired into and sealed into an exteriorpackage, the complete assembly is commonly referred to as an integratedcircuit chip, or an IC chip, or a packaged chip, or just a chip. One hasto be careful with the terminology, such as with the phrase “siliconchip”, which usually means the completely packaged part, but in othercases might refer to just the bare die. Another case is multi chipmodule, which multiple bare die are wired into and sealed into a singlepackage.

Definition of term: multi chip module

A multi chip module is the term used to describe a package of integratedcircuits in which the package contains more than a single bare die. Thusthe use of the term chip in the phrase is misleading. Usually thepackage contains a miniature printed circuit board that the individualbare die are wired to. In this way, the internal die can be pre-wired toeach other as needed, reducing the number of external wires on thepackage. Modern 3D stacking technology allows one or more bare die to bestacked vertically on top of another, or via an interposer layer. Inthis case the interposer layer replaces the function of the miniatureprinted circuit board.

IX.A. Material Considerations

There are issues of construction material properties in building acontact lens display. On one hand, the overall contact lens must havegood oxygen permeability, but on the other, the surfaces of the opticalelements and their placements have to be ridged. In one extreme, it ispossible that some amount of flexing can be tracked in real-time andcorrected for (preferably by rendering, but also potentially bymechanical effects such as piezoelectric and/or MEMS). An easierapproach would be to have the optical elements and connections betweenthem made out of a fairly ridged material, but with holes in the ridgedmaterial, and then have the rest of the contact lens be constructed froma less ridged but more oxygen permeable material. The ridged portion,which we will refer to as the display mesh, will then also serve as astructural skeleton that will keep the less ridged surrounding materialfrom flexing much. So in this case, the optical components that areconstructed from the ridged material are the optical components of thefemto projectors, but do not include the traditional functional opticalcomponent of the contact lens itself, e.g. the correction for normalvision, as seen through the display mesh. In the preferred embodimentbeing described here, the display mesh also contains the femto projectordisplay die that are the light emitting image element portion 7510 ofthe femto projectors, as well as other support chips.

IX.B. One Possible Solution: The Display Mesh

An example display mesh is shown from above in FIG. 80. The 43individual femto projector cylinders are shown as the 43 circularelements 8010 in the figure. FIG. 81 is the same design, but now the ICdie have been added in the outer ring area outside the optical zone ofthe contact lens. Element 8110 is the display controller IC die.Elements 8120 are multiple copies of the Power and Data receiving ICdie. Elements 8130 are multiple copies of the Data transmitting IC die.Element 8140 is a multi-axis accelerometer. In the preferred embodiment,the accelerometer is three axis positional and/or three axis orientationsensor, and are generally MEMS based. Sometimes such clusters alsoinclude gravitational (down) detectors, as well as magnetic (north)detectors.

Tracking the Contact Lens

It is vitally important that the location and orientation of the contactlens on the eye be accurately and rapidly tracked in real-time. To helpachieve this, some form of fluidal marks can be incorporated into thecontact lens display. Ideally, the fluidal marks should be of such adesign that they can be both rapidly and highly accurately located byimage capture (either linear or array cameras). Examples of such designsare well known to those skilled in the art. In one embodiment, thefluidal marks would be manufactured onto the top surface of the displaymesh. These “marks” can be anything that changes the way light(including infrared light) reflects or refracts from the contact lensdisplay. One embodiment of such marks would be light absorbing or lightreflecting marks (paint, etc.). Another would be optical indentation,optical extrusion, etc. In the preferred embodiment, the display meshstructure features several strategically placed corner reflectorsinscribed into its top surface. Corner reflectors have the advantagethat they reflect the vast majority of the light illuminating them backin the same direction as the illumination came from. This allowsexcellent signal to noise, e.g. a small amount of light will stillreflect back a strong point of light, a large amount of light willreflect back an extremely bright point of light. This not only allowssmaller (and thus less costly) pixels to be used on the image sensor, italso allows said image sensors to work at a very high frame rate (andstill achieve good contrast), e.g. a frame rate of 1,000 Hz or more.Examples of where fluidal marks might be placed on the display mesh areshown in FIG. 82, elements 8210. A zoom into a single corner reflectoris shown as element 8220. It is important to note that whatever theplacement of the fluidal marks is, it should not be symmetrical to thedisplay mesh, as otherwise it would be impossible to determine which ofseveral possible symmetric rotations of the contact lens had occurred.An alternative to a reflecting mark of a point design would be one of aline design. This allows a larger number of sample points from the markto be captured by 2D image sensors, or less expensive line sensors tocapture at least some portion of the line. Whatever form of fluidalmark, the idea is that the entire area of each eye where the contactlens display might be would be illuminated (preferentially by infraredlight) from the eyewear (or similar device), and image sensors would beplaced on the eyewear (or similar device) where they can observe anyregion that the contact lens can be. In the preferred embodiment inwhich corner reflectors (or something similar for line elements) areused, then the image sensor or sensors for each eye would be placed verynear, or mixed optically at, the location on the eyewear where the pointof illumination was placed. In the preferred embodiment, not only wouldan infrared light source be used, but it could be the same infraredlight source that is being used to (wirelessly) power the contact lensdisplay (and thus fairly bright), which preferably also is the sameinfrared light source where a portion of the light intensity is beingmodulated at a high frequency to provide the digital data (wirelessly)the contact lens display. Alternately, a substantially differentfrequency of infrared light than is being used for reasons other thanilluminating the fluidal marks can be dedicated for the use ofilluminating the fluidal marks, and narrow band pass optical filters canbe used in various places on both the contact lens and the eyewear toallow only the specific frequency of infrared light of interest to passas necessary.

More Display Mesh Geometric Detail

FIG. 83 shows both a top (element 8310) and side (element 8320) view ofthe display mesh. To better understand the structure and placement ofthe femto projectors in the display mesh, a side view central slice viewis shown in FIG. 78. The outer edge of the bulk contact lens is shown aselement 7810, three foveal femto projectors are shown as cylinders 7820,and six peripheral femto projectors are shown as cylinders 7830. FIG. 79is the same slice view, except this time we have sliced the femtoprojectors themselves in half, showing a simplified depiction of theinternal optics in black, as elements 7910 for the foveal and 7920 forthe peripheral femto projectors.

IX.C. Wiring Up the Display Mesh

An important concern in building an electronic device of this scale ishow the individual IC die are wired together. Technically, thecollection of die and the display mesh together can be viewed as a formof a hybrid-chip e.g., one “chip” with multiple internal die, andinternally wired together. Most hybrid chips of the past were notoptically clear, but some have included both opto-electronic die andelectronic only die. Also, usually hybrid chips have some external pinsfor power and data. In the case of the femto projectors, the power andincoming data is received optically as (in the preferred embodiment)infrared light, and data is transmitted back out of the display meshalso as infrared light. (Several alternate means are possible forpowering the device, including leaching mechanical energy intoelectrical energy off of eye blinks.) But, internally, the individualdie have some wires running between them. To reduce the number of wirestransferring data and clock from the display controller die to theindividual femto projector display die, in the preferred embodiment8b/10b encoding of both clock and data onto a single pair ofdifferentially driven wires is used. This means that there at most hasto be a maximum of two wires dedicated to communicating data and clockfrom the display controller die and each femto projector display die.There is also a requirement for the display controller IC to be able toreceive certain data back from the femto projector display die, butbecause this read-back channel has low bandwidth requirements, a singleshared differential pair of wires can serve as a bus from two attachmentpoints on the display controller die to all femto projector display die.Similarly, power and ground can be bussed to all femto projector displaydie, reducing the number of wires required. In the preferred embodimentdescribed so far, this would mean that each femto projector display diehas a maximum of six wire attachment points. (These are not called“pads,” as that terminology implies a certain scale and structure ofmaking wire attachments.) Since the display controller die is on theouter non-optical zone ring, but the individual femto projector displaydie are scatter around the optical zone connected by some common struts,this limits the paths that the wires can take. The scale of the wiringof the die is small. For example, the foveal femto projector display dieare less than half a millimeter in diameter, the peripheral femtoprojector display die are less than a third of a millimeter in diameter,and the struts are even narrower (e.g., on the order of a tenth or atwentieth of a millimeter in width). This means that the total number ofwires that can be run along a given strut must be limited, as the wirepitch of wires that can be imprinted here is ten to twenty microns ormore at their smallest.

Example Display Mesh Wiring

To see how this can be dealt with, FIG. 84 shows wire bundles 8410 fromthe display controller die to femto projector display die on separatestruts of the display mesh. The advantage of this wiring scheme is thatonly the wires going to the three, four, or in one case, five femtoprojector display die along each individual strut need be wired to onthat strut. This is a considerable reduction over running all the wiresover one strut, which, even using multiple layers of wiring, might bebeyond current wire circuit fabrication technologies. FIG. 86 is azoomed view of how the wire bundle might be decomposed into individualwires that have to connect to four femto projector display die. Toreduce the number of wires required on each strut, it is also possiblethat, rather than a separate pair of differentially encoded data wiresfor every femto projector display die, the relatively low data speedsrequired can allow three, four, or five femto projector display die tobe connected by a shared bus of just two wires. There are alsotechniques of providing both power and ground over the same data buswires, saving another two wires per strut if required. One example ofhow the return data bus from all the femto projector display die to thedisplay controller die might be wired over the struts is shown in FIG.85. The two wire bus 8510 (shown as a single line for clarity) gathersthe return data from all the femto projector display die in a tree thatkeeps the wire per strut requirement for this bus to the minimum of twowires. There are also techniques of reversing the data flow on the twodata wires going into each femto projector display die from the displaycontroller die into data outputs going back to the display controllerdie. Because the return data rate is much lower than the input data rateto the femto projector display die, the additional impedance that eachfemto projector display die will encounter when trying to drive backalong the same wires should not be too much of an issue.

Summarizing, the maximum number of wires required to be present on eachnarrow strut of the display mesh is at a minimum two, which would haveto carry the incoming, back going, and power for at most five femtoprojector display die. If the power bus is separated, then the requirednumber of wires per strut is increased to four. Further decoupling willadd more such wires with the benefit of somewhat lower power consumptionand higher maximum data rates.

IX.D. Fabricating and Assembling the Display Mesh

Each femto projector in the display mesh (43 in the current preferredembodiment) requires two quite small optical lenses, as well as anaccurate housing for the femto projector display die, and an opticalexit window for the display. The rest of the display mesh consists ofmultiple struts and the outer ring for placement of additional IC die.It would be advantageous if all these pieces don't have to be separatelyfabricated and assembled together. In the preferred embodiment, all ofthe display mesh optical, structural, and mounting portions arefabricated out of just four (or six) single piece layers. To show whymultiple layers may be needed for fabrication, examine FIG. 76. Thisshows just a single femto projector fabricated from four separate piecessuch that each piece can be fabricated using standard injection moldingtechniques, e.g., no undesired holes or voids. Contrast this to FIG. 75,which shows a completely assembled femto projector. Because the volumeis closed, it has internal (required) voids. There is no way in whichsuch a shape could be fabricated using traditional injection moldingtechniques, whereas the four separate pieces of FIG. 76 can be. Theidea, however, goes further. The entire display mesh, struts andnon-optical zone die mounting region and all can be fabricated out ofjust four separate parts, again just using standard injection moldingtechniques. This is not at all infeasible, as the same (generallyplastic) materials that are used to make high quality lenses of varioussizes are also quite suitable as structural members and die mountingareas. In FIG. 76, it can be seen what the four layers wouldindividually consist of. The topmost layer, element 7610, would form thetop of the display mesh, and the underside of it would be the die attachand wire attach surface for the electronics, both the individual femtoprojector display die, and the outer die attach ring. The outer dieattach ring is the portion of the underside of the first layer of thedisplay mesh outside the optical zone of the contact lens. This area iswhere the display controller die, several instances of power and datareceiver die, and several instances of the data transmitter die allattach. The topmost layer would also contain the uppermost portion ofthe struts between the other elements. The same will be true for thethree other layers. The layer second from the top, element 7620, withinthe femto display areas would form the first (positive) lens of thefemto projector optics, as well as part of the cylindrical sides of theprojector. The layer third from the top, element 7630, within the femtodisplay areas would form the second (negative) lens of the femtoprojector optics, as well as more portions of the cylindrical sides ofthe projector. The lowermost layer, element 7640, would form theencapsulating bottom of the display mesh.

After fabrication, the various IC die would be attached to the undersideof the top layer 7610, and then wired together on the same layer, usingthe underside of the first layer of the structural struts as paths alongwhich portions of the wiring can be run. Then all four layers of thedisplay mesh can be fastened together, using any of various techniques(e.g., various glues). Alternately, the first three layers, whichcontain no electronics or wiring (in some embodiments), could bepre-assembled, only waiting for the top assembly to be completed beforeassembling the top layer to the other three. It is advantageous that allof the outer surfaces of the assembled display mesh be “blackened” tolight, except for the optical output portion of each femto display onthe bottom of the display mesh, and the optical data/power input/outputportions above much of the top of the non-optical zone ring. This is tokeep stray light from the outside world from getting refracted orreflected by the display mesh itself, potentially lowering externalworld image quality, as well as to keep such outside light from somehowmixing in with the femto projector display optics. In some cases, theknown critical angles of reflection of a specific design and materialsmay reduce the regions at risk. It is also important that the cornerreflectors formed into the top surface of the display mesh are fullyfunctional, which might require not adding (or later removal of)blacking from these areas.

After the display mesh has been assembled (and potentially tested atvarious interim points), the “bulk” contact lens can be formed aroundthe display mesh resulting in the final contact lens display. The “bulk”contact lens material is most advantageously a plastic material withhigh oxygen permeability that could be formed by some technique aroundthe fully assembled display mesh without harming the display meshitself. The existing materials that “soft” contact lenses (e.g., siliconhydrogels, polyacrylamide hydrogels) are formed from is one likelypossibility. Because the display mesh will act as an internal skeletonto the softer material, the display mesh should cause the soft materialaround it to act in a much more ridged manner, potentially acting morelike the “hard” central portion of a soft edge scleral contact lens, orlike a more traditional “hard” contact lens. The advantage of thecentral portion of the contact lens being hard is that then astigmatismcan be automatically corrected for. The advantage of the edges of thecontact lens being softer and more pliable is the potential for a morecomfortable fit, more like that of a soft contact lens, or of a softskirt hard center scleral contact lens.

While current in production injection molding techniques allow opticalsurfaces to be routinely fabricated to better accuracy than onewavelength of (visible spectrum) light, there are constraints on howthin regions of the molded object can be. The reason is not finalstructural strength of the molded object, which still can be quitestrong over any given area, but the stresses incurred by the newlymolded object as the molds are pulled apart. There are, however, somealternate techniques for accurately forming the layers of the displaymesh out of a different optically clear material through the use ofvapor deposition. Potential materials that could be used here includediamond, silicon nitride, titanium dioxide, silicon dioxide (quartz),and aluminum oxide (sapphire). Front half molds of the negative of thedesired optical surface can have the material vapor deposited on themuntil a sufficient thickness has built up. These molds can be made ofany of a large choice of materials, including refractory metals. Thenthe inaccurate rear side of the single piece of material that was justcreated can be polished to a smooth (slightly curved) surface by adiamond sander. However, because this technique allows only one non nearplaner optical surface to be fabricated per piece, six pieces arerequired to be fabricated to together form a complete display mesh. Thisis illustrated in FIG. 77, where the optics of a single femto projectorhave been separated into six pieces that otherwise meet the vapordeposition fabrication technique. The top layer, element 7710, and thebottom layer, element 7760, are essentially the same as their injectionmold counterparts 7610 and 7640. The second layer, element 7720, nowforms the top half of the first (positive) lens of the femto projectoroptics, the third layer, element 7730, forms the bottom half. The fourthlayer, element 7740, now forms the top half of the second (negative)lens of the femto projector optics, the fifth layer, element 7750, formsthe bottom half. The use of diamond as the construction material for thedisplay mesh has several advantages. It is extremely structurallystrong, which means even when fabricated to very thin widths, it isunlikely to break during subsequent assembly or forming of the “bulk”contact lens around it. Diamond and the other materials mentioned havehigh indexes of refraction, making them good optical materials. Andunlike plastic materials, which all have some water permeability,diamond and the other materials when glued together creates a hermeticseal, eliminating the possibility of water vapor forming inside thelenses (assuming a zero humidity environment when the portions of thedisplay mesh are fastened together (argon is one such example)), andalso eliminating the possibility of condensed water possibly shortingtogether inter-die wiring. The die themselves in any case would mostlikely be pre-passivated with an optically clear hermetic seal layer.

Some alteration of the IC dies before construction will be required aswell. Normal IC die are quite thick, e.g., half a millimeter or more.That's as thick as the entire contact lens! However, this thickness isnot needed for any functioning of the IC, but for structural integrityof the wafer during IC fabrication. After fabrication, “wafer thinning”techniques can be used to make the die 20 microns or less in thickness.(In the following, remember that all die are attached upside down to thebottom surface of the first layer of the display mesh.) A related pointis that some die have to be mounted processed side up, pointing out ofthe contact lens (such as the power & data receiving die, and the datatransmitting die), while other die are required to be mounted pointingdown, such as the femto projector display die. Depending on how wiringconnections are made, this may require one or the other of such groupsof die to be able to make wiring connections on the back (non-processed)side of the die. This fits well with wafer thinning, as thinned die canhave electrical connections placed on the back of the die (via deepvias).

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the invention but merely asillustrating different examples and aspects of the invention. It shouldbe appreciated that the scope of the invention includes otherembodiments not discussed in detail above. Various other modifications,changes and variations which will be apparent to those skilled in theart may be made in the arrangement, operation and details of the methodand apparatus of the present invention disclosed herein withoutdeparting from the spirit and scope of the invention as defined in theappended claims. Therefore, the scope of the invention should bedetermined by the appended claims and their legal equivalents.Furthermore, no element, component or method step is intended to bededicated to the public regardless of whether the element, component ormethod step is explicitly recited in the claims.

In the claims, reference to an element in the singular is not intendedto mean “one and only one” unless explicitly stated, but rather is meantto mean “one or more.” In addition, it is not necessary for a device ormethod to address every problem that is solvable by differentembodiments of the invention in order to be encompassed by the claims.

In some embodiments, portions of the invention are implemented incomputer hardware, firmware, software, and/or combinations thereof.Apparatus of portions of the invention can be implemented in a computerprogram product tangibly embodied in a machine-readable storage devicefor execution by a programmable processor; and method steps of theinvention can be performed by a programmable processor executing aprogram of instructions to perform functions of the invention byoperating on input data and generating output. The invention can beimplemented advantageously in one or more computer programs that areexecutable on a programmable system including at least one programmableprocessor coupled to receive data and instructions from, and to transmitdata and instructions to, a data storage system, at least one inputdevice, and at least one output device. Each computer program can beimplemented in a high-level procedural or object-oriented programminglanguage, or in assembly or machine language if desired; and in anycase, the language can be a compiled or interpreted language. Suitableprocessors include, by way of example, both general and special purposemicroprocessors. Generally, a processor will receive instructions anddata from a read-only memory and/or a random access memory. Generally, acomputer will include one or more mass storage devices for storing datafiles; such devices include magnetic disks, such as internal hard disksand removable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM disks. Any of the foregoing canbe supplemented by, or incorporated in, ASICs (application-specificintegrated circuits) and other forms of hardware.

X. Areas of Note

In this portion of the application various embodiments discussed aboveare highlighted.

In-line display die and multiple optical elements.

Item A. An eye mounted display, comprised of multiple sub-displays, inwhich each sub display is formed by a flat multi-pixel light emittingdisplay element, followed by a first lens element, followed by a secondlens element, and optionally followed by additional lens elements, allparallel to the surface of the eye mounted display.

Item A.1. Item A, in which the first lens element is a positive lens,and the second lens element is a negative lens.

Item A.1.1. Item A.1, in which a third lens element performsophthalmological correction.

Item A.1.2. Item A.1, in which a third lens element performs additionaloptical aberration reduction.

Item A.1.2.1. Item A.1.2, in which the third lens element also performsophthalmological correction.

Item A.2. Item A, in which one or more lens elements can have theirposition shifted so as to place the display at different depth of fielddistances from the eye.

Item A.2.1. Item A.2, in which the induced depth of field distance ofthe display from the eye is determined by a function of the dynamicallytracked vergence angle of the two eyes.

Item A.2.1.1. Item A.2.1, in which the function of the vergence angle ofthe two eyes is one that dynamically places the display at the sameoptical depth that the vergence angle of the two eyes indicates thatnormal vision (non-presbyopic) is expecting to have to change the eye'sinternal lens to accommodate to.

Item A.3. Item A, in which one or more lens elements positioned betweenthe femto projectors can have their position shifted so as todynamically change the optical power of the corrective lens so as tochange the apparent depth of field of the physical world.

Item A.3.1. Item A.3, in which the amount of change in the power of thecorrective lens is determined by a function of the dynamically trackedvergence angle of the two eyes.

Item A.3.1.1. Item A.3.1, in which function of the tracked vergenceangle of the two eyes causes the power of the corrective lens to bringinto focus on the retina objects that are at the distance in the worldindicated by the said vergence angle.

Item A.3.1.1.1. Item A.3.1.1, in which it is used to dynamically curefull presbyopia.

Item A.3.1.1.2. Item A.3.1.1, in which it is used to dynamically curepartial presbyopia.

Item A3.2. Item A.3, in which the change in the apparent depth of fieldof the physical world is fixed such as to correct for known myopia orhypermetropia of an individual's eyes.

Item B. & Item C. Use of the locally uniform resolution mapping as amodel of the human (and primate) retinal receptor fields and thephysical organization of the human (and primate) visual cortex.

Item B. Using the locally uniform resolution mapping as a model ofproperties of retinal midget ganglion cells outside the foveal region.

Item B.1 Item B, in which the property of the retinal midget ganglioncells being modeled by the locally uniform resolution mapping is theirlocal density.

Item B.2 Item B, in which the property of the retinal midget ganglioncells being modeled by the locally uniform resolution mapping is localaverage area of the center portion of their receptor field.

Item B.3 Item B, in which the property of the retinal midget ganglioncells being modeled by the locally uniform resolution mapping is localaverage area of the surround portion of their receptor field.

Item B.1.1 Item B.1, in which local density of the retinal midgetganglion cells is used as a probability prediction of the specificintegral number of retinal cone cells a given modeled retinal midgetganglion cell will connect to create the center portion of its receptorfield.

Item C. Using the locally uniform resolution mapping as a model of thespatial organization of the non-foveal portion of the physical humanvisual cortex.

Item C.1. Item C in which the portion of the physical human visualcortex being modeled by the locally uniform resolution mapping is regionV1, V2, and V3.

Item D. Use of the locally uniform resolution mapping for variableresolution display in eye mounted displays, specifically includingcontact lens displays.

Item D. Use of the locally uniform resolution mapping to define thesizes of pixels displayed on the retina by an eye mounted display,outside the foveal region.

Item D. A eye mounted display device, consisting of multiple pixels, forwhich the portion of the display meant for outside the foveal region,the pixels have been arraigned such that the number of pixels at anygiven eccentricity is (approximately) constant, and the aspect ratio ofall pixels is (approximately) constant at all eccentricities.

Item D. A eye mounted display device, consisting of multiple pixels, forwhich the portion of the display meant for outside the foveal region,given any specified direction, the linear pixel density measured in thespecified direction divided by the sign of the eccentricity will be thesame everywhere.

Item D. A eye mounted display device, consisting of multiple pixels, forwhich the portion of the display meant for outside the foveal region, atany given eccentricity, the linear pixel density in the longitudinaldirection divided by the diameter of the circle on the unit spheredefined by the given eccentricity will be the same as the linear pixeldensity in the eccentricity direction (not as general).

Item E. In an eye mounted display device comprised of multiplesub-displays, having different sub-displays displaying to differentranges of visual eccentricities magnifying the size of their pixelsdisplayed such that the amount of magnification approximately followsthe locally uniform resolution mapping.

Item F. In an eye mounted display device comprised of multiplesub-displays, in which the pixels on individual sub-displays do not havea constant pixel size, but instead the size approximately follows thelocally uniform resolution mapping.

Item E+F.

Item G. Optical implementation of the locally uniform resolution mappingfor image and video capture onto standard sensor arrays.

An optical system imaging onto a planer image sensor array (still orvideo) such for any visual longitude in space, the visual eccentricityin space of light coming into the optical system is imaged onto the samelongitude on the image sensor, but that light that came from a givenvisual eccentricity is imaged onto the image sensor at a point at aradial distance r from the center of the image sensor such that r isproportional to log [tan [eccentricity/2]].

Item H. Image processing implementation of the locally uniformresolution mapping for image and video capture onto standard sensorarrays with standard lenses.

An image processing system taking in images from cameras and optics thatutilize the standard planer view projection, but that before processingre-samples the input image(s) such that an input pixel coming from aparticular visual longitude and visual eccentricity will be used as asample to contribute to the output value of a pixel with the samelongitude, but that has a radial distance in the image planeproportional to log [tan [eccentricity/2]]. Standard output pixelfiltering techniques will cause the input pixel to also contribute someamount to nearby neighbors of the said output pixel as well, thoughusually by a diminished amount.

Pre-distortion of display pixel shapes and sizes on the display die forvariable resolution display.

Item I. Pre-distortion of display pixels for correcting for residualimperfections of the femto projector optics.

Item I. Pre-distortion of display pixels for correcting for residualimperfections of the femto projector optics design.

Item I.1. Item I, in which the residual optical imperfections includethose of chromatic aberration.

Item I.2. Item I, in which the residual optical imperfections includethose of spherical aberration)

Item I.3. Item I, in which the residual optical imperfections includethose of keystone aberration.

Item I.4. Item I, in which the residual optical imperfections includethose of non-uniformity of image intensity across the field (include thecase of vignetting).

Item I.5. Item I, in which the residual optical imperfections includethose of pevetisal surface distortion.

Combinations of Items I1-5.

Item J. Pre-distortion of display pixels for correcting for fabricationand assembly errors.

Item J.0. Pre-modification of the intensity of display pixels forcorrecting for residual imperfections of the pixel intensities producedby the femto projector planer image display element caused byfabrication and assembly errors.

Item J.0.1. Item J.0, in which the intensity of light produced by aspecific individual display pixel for any given input digital pixelvalue differs from what was desired, and the correction is to substitutea different digital pixel value to be displayed that produces theclosest actual amount of light as was desired.

Item J.0.1.2. Item J.0.1, in which for most digital input pixel values,the amount of light produced by an individual pixel differs from thatwhich was desired by a linear function of the digital pixel value, andthe correction is to pre-multiply the desired digital pixel value by theappropriate inverse linear amount before the pixel is sent to the femtoprojector.

Item J. Pre-distortion of display pixels for correcting for residualimperfections of the retinal images produced by the femto projectoroptics caused by fabrication and assembly tolerance errors.

Item J.1. Item J, in which the residual optical imperfections includethose that cause in a shift in the retinal position of the imagedisplayed by a femto projector from what was desired, and the correctionis to apply an equal but opposite shift to the positions of all pixeldata before it is sent to the display.

Item J.2. Item J, in which the residual optical imperfections includethose that cause retinal images produced by the femto projector opticsto have a different magnification from what was desired, and thecorrection is to pre-minify or pre-magnify all the pixels of the imageto be displayed by the projector by an appropriate inverse amount.

Item J.3. Item J, in which the residual optical imperfections includethose that cause undesired keystone distortion in the retinal imagesproduced by the femto projector optics, and the correction is topre-apply a separate shift in position of each pixel such that theappropriate inverse keystone distortion is applied to the pixel databefore it is sent to the projector.

Item J.4. Item J, in which the residual optical imperfections includethose that cause an undesired chromatic aberration in the retinal imagesproduced by the femto projector optics, and the correction is topre-reverse this aberration generally by applying a separate shiftoffset to each different color of pixel before it is sent to theprojector.

Item J.5. Item J, in which the residual optical imperfections includethose that cause an undesired spherical aberration in the retinal imagesproduced by the femto projector optics, and the correction is topre-reverse this aberration.

Item J.6. Item J, in which the residual optical imperfections includethose that cause an undesired non-uniformity of image intensity acrossthe field in the retinal images produced by the femto projector optics,and the correction is to pre-reverse this aberration.

Item combo J. Combinations of items J1-6. A combined, separate for eachcolor of pixel, shift, change in magnification, and change in intensitycan correct for all the residual imperfections at the same time.

System of items I+J. In an eye mounted display comprised of a number ofdifferent femto projectors, perform a post-manufacturing calibrationtest separately to each said femto projector, capturing undesirableerrors in the retinal image produced caused by both residual errorsinherent in the optical design as well as those caused by fabricationand assembly tolerance errors, as well as errors in desired pixel lightoutput values caused by fabrication and assembly tolerance errors in thedisplay device, then during operation of the eye mounted display, foreach specific femto projector, derived from its specific calibrationdata, pre-apply the appropriate inverse imaging operation to the pixelsthat are to be displayed to that specific femto projector.

Feathering outer edges of the images produced by different immediatelyneighboring femto projectors.

Item I+item J, in which all the absolute positional shifts of pixelsdisplayed by a particular femto projector are known, as part of theoverall pre-computation of the pixel corrections to be applied inreal-time, from the optical calibration data pre-determine the overlapof the retinal image of adjacent femto projectors, for any pixel of afemto projector whose retinal image overlaps even a portion of theretinal image of a pixel from an adjacent femto projector, either turnoff that pixel, or determine an appropriate reduced intensity blendamount to be applied to that pixel, with the inverse of that blendamount to be applied to pixel or pixels that overlap in the adjacentprojector, with this per-pixel intensity reduction factor to bepre-combined with any other per pixel intensity pre-correction factors.

Variable Resolution Tiling of Hexagonal Projectors with DifferentMagnifications.

Tiling of Projectors.

Item K. An eye mounted display, comprised of multiple sub-displays,whose combined projected images on the retina completely cover theretina out to an outer edge, in which the shape of the pixel containingregion of the individual sub-displays light emitting element is either afixed shape that can tile the plane, or similar to the distorted versionof that shape produced by mapping that shape by the portion of thelocally uniform resolution mapping that applies to the region of theretina that a particular sub-display displays to, which is a shape knownto be able to tile a portion of the surface of a sphere.

Item K.1. Item K, in which the fixed shape is rectangular.

Item K.2. Item K, in which the fixed shape is hexagonal.

Item K.3. Item K, in which the fixed shape is triangular.

Item K.2.1, in which a group of centermost sub-displays are hexagonal inshape, and those sub-displays outside this group are those that have ashape similar to the distorted version of a hexagonal shape produced bymapping a hexagonal shape by the portion of the locally uniformresolution mapping that applies to the region of the retina that aparticular sub-display displays to.

Item K2.1.1. Item K.2.1, in which the group of displays that arehexagonal in shape are the seven centermost ones.

Item K2.1.2. Item K.2.1, in which the group of displays that arehexagonal in shape are the thirteen centermost ones.

Item K2.1.3. Item K.2.1, in which the group of displays that arehexagonal in shape are the nineteen centermost ones.

Item K.4. Item K, in which the fixed shape sub-displays combined imageon the retina covers at least the foveal region.

Item K.5. Item K, in which the eye mounted display is a contact lensdisplay, and in which the fixed shape sub-displays combined image on theretina covers the central portion of the contact lens such that for anynormally possible orientation of the contact lens on the cornea, and anynormally possible offset of the contact lens from the center of thecornea up to some specified limit, the combined image of the fixed shapesub-displays on the retina will always cover at least the foveal region.

Item K.6. Item K, in which image on the retina produced by eachsub-display overlaps by some amount the images on the retina produced bythe immediate neighboring sub-displays to the sub-display.

Item K.6.1. Item K.6, in which the amount of sub-projector overlap issufficient to allow pre-correction of residual errors in the projectedimage of any sub-projector due to limitations of the optical design, ordue to fabrication and assembly tolerance errors, without thesecorrections resulting in any un-Tillable gaps between sub-projectors.

Same techniques for cameras, including ones constructed on the front ofeye mounted displays. (Also, telepresence robotic pair of rotatablecamera eyes.)

Variable Resolution Rendering Hardware.

Item L. A hardware 3D computer graphics rendering system in which someportion of the pixels being rendered are rendered in a liner space thatis related to the viewing space via the locally linear resolutionmapping.

Item L.1. Item L, in which the portion of the pixels are rendered byrendering to a linear sample space that is later converted to the finalpixels.

Item L.2. Item L, where geometric graphics primitives are rendered bysome method of subdivision, tessellation, or evaluation in 3D space thatresults smaller and potentially simpler geometric primitives who'sbounds in the linear space fall below some threshold, and then can bedirectly rendered in the linear space.

Item L.1.2. Item L, where geometric graphics primitives are rendered bysome method of subdivision, tessellation, or evaluation in 3D space thatresults smaller and potentially simpler geometric primitives who'sbounds in the linear sample space fall below some threshold, and thencan be directly rendered in the linear sample space.

Manufacturing: display mesh, multiple layers, fabrication, die thinning,bulk contact lens.

Item M. Constructing a multitude of femto projectors optical paths byfabricating a small number (4, 6, 8) of doubly curved layers of elementsmeant to be stacked together to form the complete display mesh.

Within the display mesh, leaving holes in the layers between individualfemto projectors, so as to allow both light and oxygen through, thusallowing normal vision of the external physical world, as well asappropriate oxygenation of the cornea.

Embedding a display mesh, that may be both light and oxygen impermeable,within a second material that is both light and oxygen permeable, andfurther forming this second material in a shape such that it canfunction as a normal contact lens (the “bulk” contact lens).

The previous, in which the “bulk contact lens” may be made of a softmaterial that is comfortable to the cornea and inside eye-lid.

The previous, in which the display mesh acts as an internal skeletonthat will stiffen the bulk contact lens such that it will not bend so asto conform to the shape of the cornea, thus preserving the outer sideshape of the bulk contact lens, thus causing any astigmatic errors ofthe corneal surface to be by-passed, thus correcting for astigmatism.

Item M.0. Item M, in which some form of fluidal marks have been added tothe top layer of the display mesh to aid in real-time 3D tracking of thecontact lens.

Item M.0.1. Item M.0, in which the fluidal mark is formed by impressinga corner reflector into the top surface of the display mesh.

Item M.0.2. Item M.0, in which the fluidal mark is formed by aretro-reflector embedded into the top of the display mesh.

Item M.0.3. Item M.0, in which the fluidal mark is formed by addingoptically absorbing material, such as, but not limited to, ink, in aspecific target pattern, onto the top of the display mesh.

Item M.1. Item M, in which an outer ring is present outside the opticalzone of the eye mounted display, on which non femto projector die areattached and wired up.

Item M.1.1. Item M.1, in which the wiring to the femto projector imagedie follows over the bottom (or top) of the structural struts connectingthe individual femto projectors together.

Item M.1.2. Item M.1, in which the data wires to and from the femtoprojector image die along a particular strut path from outside theoptical zone to or next to the center of the display mesh carry onlydata to or from those specific femto projector image die.

Item M.2. Item M, in which all IC die to be included inside the displaymesh have been thinned using wafer thing techniques such that the dieare thin enough to fit in a prepared gap between two layers of thedisplay mesh.

Item M.3. Item M, in which the femto projector image die has beentrimmed from square or rectangular to a shape closer to circular so asto better fit within the circular optical path of the femto projectorwithout making the outer shell of the femto projector larger than itneed be.

Item M.3.1. Item M.3, in which the shape closer to circular isoctagonal, though not necessarily with all edges of equal length.

Item M.3.2. Item M.3, in which the shape closer to circular ishexagonal, though not necessarily with all edges of equal length.

Item M.4. Item M, in which each of the individual layers has beendesigned such that it can be formed by a mold with no gaps or voids.

Item M.4.1. Item M.4, in which one side can be formed by a mold with nogaps or voids, but the other side need only be formed by rotationalpolishing.

Item M.5. Item M, in which all the IC die and associated wiring areattached on one side of one layer of the display mesh.

Item M.5.1. Item M.5, in which the side that all IC die and wires areattached to is the bottom side of the topmost layer of the display mesh.

Item M.6. Item M, in which the material that the individual layers ofthe display mesh are fabricated from are oxygen permeable.

Item M.7. Item M, in which the material that the individual layers ofthe display mesh are fabricated from are not oxygen permeable, such thatwhen all the layers have been bonded together, the interior of thedisplay mesh is hermetically sealed.

What is claimed is:
 1. An eye mounted display, comprising: a contactlens; and a plurality of image projectors mounted in the contact lens,each image projector comprising: a display element that produces animage; and non-folded optics that project the image from the displayelement onto a retina of a user wearing the contact lens, where theimage projector has an in-line optical design; and where each of thedisplay elements produces a different image.
 2. The eye-mounted displayof claim 1 where the image projector has a thickness of not more than0.5 mm.
 3. The eye-mounted display of claim 1 where the image projectorhas a volume of not more than 0.5 mm×0.5 mm×0.5 mm.
 4. The eye-mounteddisplay of claim 1 where the non-folded optics are constructed from atleast two layers, each layer including a portion of the non-foldedoptics for more than one of the image projectors.
 5. The eye-mounteddisplay of claim 1 where the image projectors are cylindrically shapedand inserted into the contact lens.
 6. The eye-mounted display of claim1 where the image projectors comprise at least two different opticaldesigns.
 7. The eye-mounted display of claim 1 where the plurality ofimage projectors are peripheral image projectors, and the eye-mounteddisplay further comprises one or more foveal image projectors, thefoveal and peripheral image projectors cooperating to tile the retinawith projected images, the foveal image projectors projecting imagesonto the retina with a higher resolution than the peripheral imageprojectors.
 8. The eye-mounted display of claim 1 where the imagesprojected by the image projectors onto the retina are tiled according toa hexagonal tiling.
 9. The eye-mounted display of claim 1 where theplurality of image projectors are foveal image projectors, and theeye-mounted display further comprises multiple peripheral imageprojectors, the foveal and peripheral image projectors cooperating totile the retina with projected images.
 10. The eye-mounted display ofclaim 1 where the non-folded optics has an optical axis that isperpendicular to a curvature of the contact lens.
 11. The eye-mounteddisplay of claim 1 where the non-folded optics comprise one or morerefractive lens elements.
 12. The eye-mounted display of claim 11 wherethe non-folded optics further comprises an exit window.
 13. Theeye-mounted display of claim 11 where the refractive lens element isformed from one of diamond, silicon nitride, titanium dioxide, quartzand sapphire.
 14. The eye-mounted display of claim 11 where therefractive lens element is injection molded.
 15. The eye-mounted displayof claim 11 where the image projector further comprises a housing inwhich the display element is mounted.
 16. The eye-mounted display ofclaim 13 where the housing is a blackened housing.
 17. The eye-mounteddisplay of claim 1 where the non-folded optics comprises a negative lenselement and a positive lens element.
 18. The eye-mounted display ofclaim 1 where the display element comprises a wafer thinned die.
 19. Theeye-mounted display of claim 1 where the display element comprises awafer thinned die with a thickness of not more than 20 microns.